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TEACHING EVIDENCE-BASED DECISION MAKING 


Introduction 


LEE S. SHULMAN 


I write the introduction to this collection of papers at the conclusion of the 
historic 2008 presidential election. This election is germane to this collection 
because it highlights the responsibility of citizens to weigh the arguments and 
proposals of competing candidates and to support those whose proposals are 
most reasonable and whose reasoning is most compelling. But voting citizens 
are also evaluating the characters, beliefs, personalities, and life histories of 
candidates. Should these judgments and evaluations be pursued through crit- 
ical analyses of data and arguments, through the exercise of intuitive empath- 
ic “gut” feelings, or through some combination of both? 

Is it a worthwhile goal for every citizen in a democracy to exercise good 
judgment? If such a goal is worthwhile, can good judgment be taught? Can 
the wherewithal to think critically, analytically, and soundly be learned by stu- 
dents, whether in the precollegiate years or in colleges and universities? Can 
these ways of thinking and reasoning be taught in formal educational set- 
tings? Even if they can, are the most important determinants of good judg- 
ment to be located in developed cognitive capacities or in the development of 
good character? 

The founders of democratic societies disdained the claim that critical 
thinking and good judgment were hereditary gifts that were possessed only 
by a small, elite segment of the society. If such talents are not innate and not 
limited to a select aristocracy of women and men—if, in fact, these modes of 
thought and action are teachable—should they be treated as “basic skills” 
comparable to reading, writing, calculation, and—in our present era—funda- 
mental technological literacy? Because evidence-based decision making is 
unlikely to be learned in the family or the neighborhood, should schools not 
be responsible for teaching this skill from kindergarten through graduate 
school? The question then becomes whether teaching evidence-based deci- 
sion making really matters, whether it is possible, and how it might be 
accomplished. 

An alternate view would subordinate evidence-based decision making to 
the development of desirable values, beliefs, attitudes, or habits. Other 
virtues may indeed be far more important than critical reasoning—such as 
kindness, empathy, love, loyalty, fairness, fidelity, beauty, or faith. We may err 
if we place too much emphasis on the academic values of reason and not 
enough on those in the moral, humane, religious, and aesthetic spheres. 
Which is better: to have leaders of questionable virtue reasoning clearly and 
relying on carefully collected and weighed evidence; or to rely on fundamen- 
tally decent, sensitive, and just men and women who will use evidence imper- 
fectly but nevertheless in a humane and considerate manner? 
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This volume includes five essays commissioned by the American Academy 
of Arts and Sciences to respond to these questions. The respondents include 
three psychologists: Jerome Kagan, Richard Nisbett, and David Perkins; 

a science educator, Tina Grotzer; and a political and moral philosopher, 
Eamonn Callan. They offer quite different perspectives on these common 
questions, perspectives sufficiently distinctive that finding common grounds 
for comparison is frequently difficult. 

Three of the authors—Nisbett, Perkins, and Grotzer—gather the growing 
evidence that critical judgment, probabilistic reasoning, and clear thinking 
can be taught, whether in general or specifically in the teaching of particular 
disciplines, such as the sciences. They further argue that once learned, such 
modes of thinking can, with limits, be transferred to other domains and are 
therefore likely to lead to more general capacities to think critically and effec- 
tively. Kagan scoffs at such claims for generic critical-thinking abilities, insist- 
ing that good judgment is much more likely to be domain, discipline, and 
context specific. He further expresses doubt that cognitive capacities alone 
are sufficient. Callan shares a modicum of Kagan’s skepticism. He reflects on 
how difficult it has been for him, a seasoned philosopher and experienced 
scholar, to learn to think across traditional disciplinary boundaries. He sug- 
gests Americans should learn to reason about and embrace questions of val- 
ues and moral choice rather than learn to weigh likelihoods and the warrant 
of empirical evidence. 

Concern for whether higher education is indeed “adding value” to the 
abilities of undergraduate students has now reached the state and national 
policy arena. U.S. Secretary of Education Margaret Spellings appointed a 
commission—under the chairmanship of Charles Miller, who had led the 
Texas higher education commission—to examine the quality of U.S. higher 
education. One of the central items on the commission agenda was whether 
colleges and universities were holding themselves accountable for improving 
the quality of knowledge and thought demonstrated by their students. While 
the value added in specific professional programs, such as law, engineering, 
and dentistry, might be quite obvious, what is added by undergraduate liber- 
al education? 

Although most of the critics of the Spellings Commission attacked the 
specter that multiple-choice tests might be employed to answer that ques- 
tion, the most frequently cited example of such an assessment was not a mul- 
tiple choice test but an open-ended essay exam, the Collegiate Learning 
Assessment. This assessment of critical thinking and evidence evaluation has 
been designed by some of the nation’s premier scholars of assessment and 
cognition and seeks to measure the gains in general critical thinking and rea- 
soning abilities achieved by college students over the course of their under- 
graduate experience. 

Americans may tend to orient toward general processes of thinking and 
problem solving rather than to deep substantive understanding within specif- 
ic disciplinary domains. This, at least, appears to be the perception overseas. 
Several years ago, a group of colleagues and I met at Rhodes House at 
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Oxford University to discuss the first draft of our book Educating Lawyers 
(Sullivan et al. 2007). At a critical point in the two days of discussion, one of 
the British colleagues asserted —with substantial concurrence from his fel- 
lows —that the trouble with U.S. legal education is that its goal is to teach 
students “to think like lawyers,” to engage in the analytical reasoning associat- 
ed with the processes, strategies, and dispositions of the law rather than with 
the concepts, principles, and facts of the law—the “black letter law”—that are 
the core of legal understanding and competence. 

My colleagues and I at the Carnegie Foundation have for the past decade 
been studying education in a number of professions —law, clergy, engineer- 
ing, medicine, nursing, scholarship (the Ph.D.), and school teaching. We are 
persuaded that education for the learned professions is not a matter of intel- 
lectual development alone, though that is a central challenge of every profes- 
sional school. The candidates in these fields must also develop high levels of 
technical skill; they must learn to do, perform, and act. Even though techni- 
cal accomplishments are often disdained by the academy (note the lower 
prestige of “clinical appointments” in academic schools of medicine, nursing, 
or education), they are the attributes that distinguish the most accomplished 
professionals. Yet even technical excellence is insufficient. The well-educated 
professional must develop an internalized sense of responsibility, an integrity 
that is inherent in their identities and embodied in their thought, skills, and 
capacities for responsible judgment and action. In the Carnegie studies we 
refer to these three kinds of learning as habits of mind, habits of practice, and 
habits of the heart. I am persuaded that this three-fold representation would 
also map nicely onto the learning needed by citizens: to think and act with 
integrity is no small feat. 

The readers of these essays will find them challenging in their disagree- 
ments, but need not make an either/or choice among the several perspectives 
presented here. They are not mutually exclusive because all, in their way, pro- 
mote one of the most powerful ideas in contemporary thought, that of ‘prac- 
tical judgment’? When we say that we aim to help students develop the capaci- 
ties to reason critically and act responsibly, we may be saying that we wish 
them to learn to engage well in the process of practical reason. Whether choos- 
ing a car to buy, a candidate for whom to vote, a medical treatment to elect, 
or an engineering design to pursue, individuals reason practically as they mull 
their alternatives. They do seek and evaluate empirical evidence in connection 
with their possible choices. How much fuel does a hybrid car really save, and 
how much lower is the amount of pollution it pours into the air? What is 
more important, a candidate’s positions on Iraq, abortion, evolution, prayer 
in schools, or healthcare? Do I prefer a leader with the “right” policies but 
questionable character to one who appears honest and sincere but disturbing- 
ly uninformed about important issues? 

These are the kinds of questions that call for the exercise of practical rea- 
son, a form of thought that draws concurrently from theory and practice, 
from values and experience, and from critical thinking and human empathy. 
None of these attributes is likely to be thought of no value and thus able to 
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be ignored. Our schools, however, are unlikely to take on all of them as goals 
of the educational process. The goal of education is not to render practical 
arguments more theoretical; nor is it to diminish the role of values in practi- 
cal reason. Indeed, all three sources —theoretical knowledge, practical know- 
how and experience, and deeply held values and identity—have legitimate 
places in practical arguments. An educated person, argue philosophers 
Thomas Green (1971) and Gary Fenstermacher (1986), is someone who has 
transformed the premises of her or his practical arguments from being less 
objectively reasonable to being more objectively reasonable. That is, to the 
extent that they employ probabilistic reasoning or interpret data from various 
sources, those judgments and interpretations conform more accurately to 
well-understood principles and are less susceptible to biases and distortions. 
To the extent that values, cultural or religious norms, or matters of personal 
preference or taste are at work, they have been rendered more explicit, con- 
scious, intentional, and reflective. 

In his essay for this volume, Jerome Kagan reflects the interactions among 
these positions by arguing: 

We are more likely to solve our current problem, however, if teachers 

accept the responsibility of guaranteeing that all adolescents, regard- 

less of class or ethnicity, can read and comprehend the science section 

of newspapers, solve basic mathematical problems, detect the logical 

coherence in non-technical verbal arguments or narratives, and insist 

that all acts of maliciousness, deception, and unregulated self-aggran- 

dizement are morally unacceptable. 


Whether choosing between a Prius and a Hummer, an Obama or a 
McCain, installing solar panels or planting taller trees, a well-educated person 
has learned to combine their values, experience, understandings, and evi- 
dence in a thoughtful and responsible manner. Thus do habits of mind, prac- 
tice, and heart all play a significant role in the lives of citizens. 

Our writers offer a rich array of perspectives on the potential for the edu- 
cational process to inform and enrich this complex set of processes. I com- 
mend their offerings to the readers of this volume: you will find them stimu- 
lating, disturbing, and enlightening in turn. 
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CHAPTER 1 


Decision Making and 
Its Development 


DAVID N. PERKINS 


Review your activities over the last few weeks. Perhaps you can recall a deci- 
sion that did not turn out so well. The outcome might not have been your 
fault: Even when we ponder long and wisely, circumstances intervene—a 
hurricane, a flu, a flat tire. But people often recognize something they could 
have done to make a better decision in the first place. “If I had thought about 
it more carefully, I would have recognized that five or six things could go 
wrong, so at least one of them was almost bound to” “I could easily have 
bought some time to get more information.” “If I had considered the possi- 
ble side effects more, I would have realized that plan B was really better than 
plan A’ “T let myself be influenced far too much by what others were saying.” 
And so on. 

I get this simple exercise from a friend and colleague of many years, 
Robert Swartz, a prominent figure in the critical-thinking movement. The 
exercise is suitable for teachers, children, professors, businesspeople. In 
groups it generates remarkably rich conversations about the art and craft of 
decision making. Almost everyone can respond with at least one significant 
decision that did not go so well and how the decision might have been better 
approached by paying more attention to one or another kind of evidence. 

We should not be surprised. Our lives are woven out of decisions. We are 
constantly deciding—whether to do homework or go to a movie, whether to 
accept a job offer or keep looking, whether to call someone for a second date, 
whether to vote for this person or that, support this policy or that. Moreover, 
decision making is not one of those human capabilities like breathing or 
everyday conversation that pretty much takes care of itself. A complicated 
world and our complicated selves ensure that decision making is often hard 
and sometimes, to our later chagrin, disastrously easy. 

Looking to the larger scale, the decisions of leaders are likewise not 
untroubled. Historian Barbara Tuchman (1984) in The March of Folly system- 
atically analyzed a number of historical decisions that she argues were 
“folly’—unwisely taken at the time, with ample warning from insiders and 
outsiders. Examples include King George’s decision to play tough with the 
American colonies, leading to the American Revolution; and U.S. involve- 
ment in Vietnam. The January 1986 decision to launch the Challenger space 
shuttle despite serious concerns from engineers is another often-mentioned 
example: executive concerns about the appearance of progress inappropri- 
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ately overrode technical reservations (Starbuck and Milliken, 1988). The shut- 
tle exploded shortly after launch. The poorly conceived Bay of Pigs invasion 
of Cuba is a further common target of critique (e.g., Johnson 2004). 

Imagining a society where evidence-based decision making is more of a 
deliberate craft, where most people make decisions somewhat better than 
they typically do today—because education, mentoring, and other means of 
fostering good decision making have brought this about—is enticing. So, 
too, is imagining a society where adolescents are more cautious about experi- 
menting with sex and drugs, where adults less often fall into dead-end jobs or 
dead-end relationships, where care for the elderly involves more planning and 
foresight. So, too, is imagining a world where not only do individuals better 
navigate their way through the complexities of life but where larger-scale 
decisions — matters of public office and public policy—reflect more informed 
and attentive leaders and citizens. My aim in this essay is briefly to explore 
the prospects of creating such a world. 

Focusing on everyday decision making, the discussion moves through 
several themes: first, what distinguishes decision making from other kinds of 
thinking; then, descriptions of how decision making goes wrong and pre- 
scriptions for what counts as good decision making and how education 
might develop good decision making; finally, a particular problem: how bet- 
ter decision making might serve not just individual interests but civic engage- 
ment and society at large. 

Research tells us a lot today about how decision making goes right or 
wrong. What we know adds up not so much to a template for perfect deci- 
sion making as to a vision of artful self-regulation responsive to the circum- 
stances, sometimes more analytic, sometimes more intuitive, sometimes mak- 
ing checklists, sometimes telling stories and counterstories to oneself. Mode 
needs to match occasion. The best efforts to improve decision making 
through education have beneficial results. However, these efforts focus only 
on some aspects of the challenge. Educational and other institutions would 
do well to invest much more in this fundamental cognitive craft through 
which we shape our lives and our larger society. 


DEFINING DECISION MAKING 


How does decision making differ from other kinds of thinking, for instance, 
planning, reasoning, or analyzing? The boundaries between such everyday 
categories inevitably are fuzzy, all the more so because they tend to draw on 
one another— one decides on a plan or plans how to decide, reasons out a 
decision or decides on the most important reasons (e.g., Moshman 2005). 
That acknowledged, decision making most centrally is a problem of choice 
(Galotti 2002). One must select from among alternative paths leading toward 
the future. The paths may comprise alternative plans, go/no-go choices, or 
beliefs to accept or reyect—as with religious faiths, political philosophies, sci- 
entific theories, or personal policies. 
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Often the challenge goes beyond deciding between given alternatives. 
One might need to find reasonable alternatives in the first place or to look 
creatively beyond the obvious options to others more promising. Even so, 
choice remains central. In the search for a good choice, many factors can fig- 
ure: goals and values to be pursued, time pressure, side effects, opportunity 
costs, stable versus changing circumstances, available knowledge, interests of 
others as well as oneself, and more (Byrnes 1998; Galotti 2002; Jacobs and 
Naczynski 2005). 

While decision-making situations vary enormously in their particulars, 
this account centers on what might be called everyday decision making—for 
instance, purchase decisions; forming and severing personal relationships; 
voting and other political commitments; educational and job decisions; and 
choices regarding sex, recreational drugs, and the like. People addressing such 
everyday decisions typically face possibilities of action rather than, or in addi- 
tion to, belief; /ave time for reflection rather than needing to respond quick- 
ly; have some familiarity with the area in question but lack deep professional 
expertise; and feel they are personally involved rather than making technical 
decisions on matters not close to them. 

An ordinary example: A couple of years ago, I attended a conference in 
Athens. My wife came along and we took time to enjoy the classic city. On 
our last day we spent an hour-and-a-half looking at Oriental rugs in a small 
shop. We spent considerable thoughtful time sorting out which rugs we liked 
best and ended up buying three medium-size rugs. We knew something 
about the pricing of these rugs, and the costs seemed reasonable. Our deci- 
sion, like countless others that people make every day, had all the earmarks of 
a typical decision-making process: an action taken after reflection, based on 
some familiarity with the area, and having personal involvement. 

The decision also was a mistake, we later concluded: not a serious one, 
but a mistake nonetheless. Why? How do decisions go wrong? 


CHALLENGES OF DECISION MAKING 


The next day we felt that we should have bought only one or perhaps two of 
the rugs. Our imminent departure had led us to overpurchase. Leaving 
behind an attractive buy was difficult. An accident of schedule—our immi- 
nent departure—unduly influenced us (the shop owners did nothing to press 
us) in a way that advertisers frequently mimic with headlines like “One day 
only” or “Limited time offer” Amsel et al. (2005) argue that avoidance of 
future regret often strongly and inappropriately figures in decision making. 
In effect, my wife and I feared the regret of a lost opportunity, a decision- 
making pitfall we should have recognized from prior experience. 

Were we sure we made a mistake? This is an important question because 
appraisal of decisions after the fact suffers from what is called “hindsight 
bias,” the distorting effect of knowing how things turned out (e.g., Fischhoff 
and Beyth 1975; Teigen 1986). We were confident we had erred. We recalled 
how during shopping we were much more attracted by one of the rugs, sig- 
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nificantly by the second, and found the third simply pleasant. This should 
have been a signal for caution. After all, the rugs were being shipped, so we 
could have taken a couple of days to decide about the second and third rugs 
and then e-mailed the shop owner with little realistic risk that others would 
have purchased exactly those rugs. We could even have photographed the 
rugs with our digital camera for another look later. We had done both these 
things in the past. 

To generalize from this and many other ordinary examples: decision mak- 
ing, broadly speaking, goes wrong because it is not sufficiently evidence 
based. People often act impulsively and impressionistically, with a limited 
view of the matter at hand, not looking beyond what the obvious options 
offer; not sufficiently consulting past experiences, general knowledge, 
informed friends and colleagues, and other sources; or not weaving evidence 
together in a coherent way. The technical literature on the psychology of 
decision making foregrounds how poor decision making reflects lack of self- 
regulation to elaborate and clarify one’s goals and the goals of relevant others, 
imaginatively search for promising options, carefully assess the costs and con- 
sequences of adopting those options, and so on (e.g., Baron 1988; Byrnes 
1998; Galotti 2002; Janis 1989). 

That is the big picture. Now for some details. 


The Challenge of Representing Complexity 


One obvious challenge of decision making is simply that decisions are often 
complex, involving several options, multiple criteria, and diverse knowledge 
about the situation. Managing all the elements is hard, and overlooking 
something important is easy. Means are needed to organize and integrate the 
ideas and information important to a decision. 

The good news is that people have conceptual tools for coping with this 
complexity. The tools are not always fully or well used, but they are powerful. 
Two rather different modes of representation figure prominently in both 
everyday and more expert decision making: guantitative/tabular (quantitative 
for short) and narrative. 

The quantitative approach often takes simple list form, as in checklists of 
product features or tallies of pros and cons. At the far end of technical finesse 
are expected-utility models. These involve an intricate tabular analysis, with 
each of several options scored for each of several criteria and the criteria 
weighted for relative importance (e.g., Baron 1988; Dawes and Corrigan 
19743 von Winterfeldt and Edwards 1986). Multiplying each option score 
times the weight of the criterion and summing the results yields an overall 
score for the option. The probabilities of various consequences can also be 
incorporated into the analysis. Impressively sophisticated as this is, it is a 
method that does not serve all situations well, a matter elaborated later. 

Narrative offers another mode of representation for decision making. 
Simple versions of the narrative mode involve concise causal stories in favor 
of or against options: “Pm sympathetic with his goals, but okaying this 
would set a precedent we'd have to live with forever” “If we order today, 
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we'll have to have the materials sent by express or they simply won’t get here 
in time.” Complex versions involve elaborate argument. In criminal law, for 
instance, consider how the prosecution motivates the jury to return a guilty 
verdict by weaving the evidence into a story, a “theory of the crime” that 
combines the three themes of motive, method, and opportunity. This story 
must stand against counterstories advanced by the defense, which cast the 
evidence in a different light and challenge some of it. 

While the conceptual tools of quantitative and narrative representation 
can help with decision making, their ready, widespread use is complicated by 
a number of factors. For decisions that invite technical approaches, the more 
intricate analyses require knowledge and skills many people do not have. On 
the quantitative side, most people do not know when to attempt or how to 
manage the expected utility procedures. On the narrative side, most people 
are unlikely to be conversant with the niceties of hypothetico-deductive rea- 
soning. However, lack of technical knowledge is probably not the principal 
mischief maker. A considerable amount of decision making is simply shallow. 
People do not try hard enough to be better informed, to consider multiple 
possibilities, and to carefully evaluate them. Why is this? 


The Trend to Oversimplification 


Benign factors lurk behind many quick and dirty decisions. Occasionally 
choices arise in urgent circumstances. With no time for elaborate analysis, 
one must take one’s best shot, which can in fact be quite good. Other benign 
factors include the fact that investment in a complex analytic process can be 
costly, many decisions are not all that important, and the prospective gains 
from taking the longer, more thorough route might not outweigh the 
prospective benefits of available shortcuts. For important decisions, the avail- 
able information may not support the most thorough styles of analysis any- 
way. Figuring in relative probabilities of consequences will add minimal value 
if the probabilities are little more than guesses. 

On the other hand, many not-so-benign factors lead to shallow explo- 
rations of decisions. A pointed folk saying —“Between two options, choose 
the third”— points to one of the most basic causes: important options beyond 
the obvious ones often exist. The same can be said of criteria. Many decision 
situations do not make salient some of the promising options and pertinent 
criteria. The situations create the illusion of a completely stated problem: a 
couple of apparent choices and two or three obvious considerations. 
However, a little digging reveals hidden layers. 

This was true of the rugs story. Our focus on which made overlooking 
deferred purchase strategies easier. Similarly, given the option of buying a 
neighbor’s used car, many people will assume their choices are yes or no after 
a test drive. But perhaps they could “choose the third” and make a deal to 
rent the car for a week to see how they like it, the rental going toward the 
purchase price if they’re happy. 

Problems of saliency aside, elaborate decision making demands intense 
mental effort. Herbert Simon (1957) famously characterized the problem of 
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human beings’ Limited rationality: cognitive bottlenecks steer us toward expe- 
ditious but sometimes unfortunate shortcuts. Touching on the rugs purchase 
once more, my wife and I had a lot on our minds and little room for much 
more. Analogously, a manager focusing on the substance of a tricky decision 
can easily forget a colleague whose judgment should be heard both for coun- 
sel and to maintain the relationship. 

Keeping things simple is a natural and generally adaptive tendency. 
Contemporary “dual processing” models argue that people have two modes 
of processing: intuitive and analytic’ (e.g., Epstein 1994; Klaczynski 2005; 
Reyna et al. 2005; Stanovich 1999; for a critical assessment, see Evans 2008). 
Intuitive decision making is rapid, is only marginally conscious, and at its 
best is deeply informed by experience in the domain in question. Imagine 
how you might size up a job candidate in the course of a brief conversation, 
responding to all sorts of nuances you could not readily name. In contrast, 
analytic decision making is more conscious, deliberate, effort-demanding, 
systematic, and analytical, pondering a number of options and integrating a 
range of considerations toward a conclusion by using narrative, quantitative, 
or mixed approaches. 

Efficient, fairly intuitive processing dominates our day-to-day activities— 
we would need a month to get through a day otherwise—and also many 
medium- to high-stakes decisions. Often this serves well enough, but some- 
times the limited horizons and technical shortfalls of intuitive processing 
leave us blind to significant options, benefits, and risks. Moreover, people 
sometimes deceive others and even themselves about their commitment to 
explicit evidence, formulating articulate rationales that are no more than post 
hoc justifications of intuitive convictions. 


The Hazards of Heuristics and Biases 


Intuitive processing is replete with what are sometimes called heuristics and 
biases. These rules of thumb often serve in a rough-and-ready way but can 
prove deeply misleading (Kahneman et al. 1982). Sometimes characterized as 
cognitive illusions, these include, for instance, confirmation bias (the tenden- 
cy to seek confirmatory and fail to seek disconfirmatory evidence), social 
stereotyping, the fundamental attribution error (the tendency to explain oth- 
ers’ behavior as reflecting abiding personality characteristics while seeing 
one’s own behavior as a response to situational demands), the neglect of 
opportunity cost (the cost incurred by foregoing the benefits of A when you 
choose B over A), and the “waste not” heuristic around sunk costs (the feel- 
ing that you have to get your money’s worth out of a cost already irretriev- 
ably incurred, even though a change in circumstances, or something else, 
may have made the effort of doing so no longer worth the gain). 

Another important bundle of biases, termed prospect theory, won Daniel 
Kahneman the 2002 Nobel Prize for economics (his partner in this work, 
Amos Tversky, unfortunately had died or he would have shared the award) 


1. Other terms are sometimes used. 
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(Kahneman and Tversky 1979). Prospect theory focuses on decisions that 
involve gains and losses, often under conditions of uncertainty. 

The theory has several nuances, but one of the most basic points is easily 
stated: People are strongly loss-averse—changes viewed as losses usually are 
treated as considerably more painful than equivalent changes viewed as gains 
are treated as rewarding. Contrary to Ben Franklin, a penny saved is worth 
more than a penny earned. A $100 bill is worth more to you, by roughly 
twice as much, when you have it and think of yourself as potentially losing or 
spending it than when you don’t have it and might gain it. For instance, peo- 
ple usually won’t bet $100 against a fifty-fifty chance of getting $X until X gets 
up to about $200 (e.g., Tversky and Kahneman 1992). 

To see how losing versus gaining works in a nonmonetary case, imagine 
that Tony is more of a Superman than a Spiderman fan and Patrick the oppo- 
site. Unfortunately, Tony receives a Spiderman action figure on his birthday, 
and Patrick receives a Superman action figure. Will they trade? Probably not, 
because once Tony possesses a Spiderman figure, trading it away gets coded 
as a loss, likely to seem too dear compared to gaining a Superman action fig- 
ure unless the initial preference difference is very great. 

Such differences reflect what is called a framing effect: losses and gains are 
counted relative to the baseline from which decision makers see themselves as 
starting. Framing effects are errors of reasoning because the underlying situa- 
tion is the same no matter how framed. Tony would enjoy the Superman fig- 
ure more and Patrick the Spiderman figure more if only they could get past 
their loss aversion and make the trade. 

Framing effects also influence risky situations. People tend to be risk- 
averse about gains, preferring a smaller sure thing to a chance at a greater 
gain; and they tend to be risk-seeking about losses, preferring a risk of a 
greater loss to a certain smaller loss. One notable study asked a number of 
ambulatory patients, graduate students, and physicians about their preference 
between two different treatments for lung cancer (none of the subjects actual- 
ly suffered from the malady) (McNeil et al. 1982). For about half the subjects 
the results of the two treatments were expressed in gain terms (e.g., survival 
rates at various points in time), and for the rest in loss terms (e.g., mortality 
rates). One of the treatments involved a higher risk of early death but greater 
long-term survival. When the treatments were described in terms of survival 
rates, subjects tended to shun the risky “early death” treatment; they tended 
to choose it when the treatments were described in terms of mortality rates. 

In principle, people might notice that the treatments could be described 
either way and thus more objectively think through the matter. After all, the 
job of analytic processing is to monitor the output of intuitive mechanisms 
and catch shortcuts when they seem likely to do serious mischief. One prob- 
lem, though, is that analytic processing inevitably misses a lot. Constant 
checking would require far too much of our cognitive resources. 

Another problem is that analytic processing generally is unaware of spe- 
cific glitches such as loss aversion or sunk costs. For instance, proceeding in a 
broadly analytic way, one might happily register a sunk cost as a real loss in a 
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pro-con list. Sophisticated analytic practice involves not only an elaborated 
quantitative or narrative approach but alertness to specific traps. 


The Neglect of Intuition at Its Best 


Without question quick, intuitive decision making serves well to move us 
through the minor decisions of the day. When the stakes are low, and even 
more so when we have experience to build on, why think hard about a deci- 
sion? If we are occasionally wrong, so what? We can hope to learn from our 
mistakes and do better next time. 

However, for more important decisions, the moral of the story so far 
might seem to be “analytic: good; intuitive: bad” To avoid Simon’s limited- 
rationality bottleneck, decision makers should fight the intuitive impulse and 
adopt well-elaborated narrative and quantitative approaches with the help of 
support systems from pencil-and-paper to spreadsheets. 

A significant body of research, however, shows that sometimes intuitive 
decisions ave better. Malcolm Gladwell’s (2005) popular book Blink reviews a 
range of such examples, albeit somewhat uncritically. In general, intuition can 
serve well when the decisions have a personal character, for instance prefer- 
ences for food, art, or perhaps even another person to spend one’s life with. 
For example, in laboratory experiments where subjects were invited to 
choose a jam they liked or a poster to hang in their apartments, subjects asked 
to approach the task intuitively showed greater satisfaction later on with their 
choices than did subjects asked to take an analytic approach (Wilson et al. 
1993; Wilson and Schooler 1991). 

Why is this? Often we do not have precise access to our reasons for liking 
something. More broadly, in some areas of life we cannot simply read off the 
tops of our minds the many tacit goals that guide our behavior. Adopting an 
analytic approach can disrupt our intuitive attunement to our deep interests, 
leading to worse decisions. (These tacit goals may themselves invite surfacing 
and reconsideration, but that is another story.) 

Research also argues that an intuitive approach often better serves when 
people have well-developed experience in a domain. Studies of experts have 
yielded a model called recognition-primed decision making (RPD) (e.g., Wein 
1998; Lipshitz et al. 2001). Experienced firefighters, ship commanders, tank pla- 
toon leaders, and so on tend to make decisions by sizing up a situation rapidly 
in order to understand it. From this, a good prospective solution quickly and 
naturally emerges. When the expert has doubts, instead of brainstorming many 
solutions and comparing them, he or she typically adopts a narrative tactic of 
limited scope, assessing the solution at hand by doing a mental simulation and, 
if the solution seems wanting, reaching for another prospect. Especially in time- 
pressured circumstances or where efficiency is important, the point is not so 
much to choose the absolute best among all possible options as it is to arrive at 
a good workable solution and move forward. 

RPD should not be seen as limited to professional expertise. In many of 
the ordinary activities of life—choosing a smart alternative route when you 
encounter a traffic jam on your usual commute, deciding whether to try to fix 
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a plumbing problem yourself or call the plumber—RPD may well figure 
effectively. When my wife and I were buying Oriental rugs in Greece, we had 
enough experience to recognize a risk of overpurchasing. Perhaps next time 
our recognition will be better primed! 

Does this suggest a moral directly opposite “analytic: bad; intuitive: 
good”? Do we just need to develop enough knowledge and experience in the 
domain in question to underwrite intuitive decision making? As a general 
policy this will not work. Many life decisions occur only now and then, with 
most people unlikely to accumulate the rich expertise that could drive sound 
intuitive decision making. 

Moreover, considerable research shows that when multiple reasonably 
clear criteria bear on a decision people are not good at intuitively integrating 
their impressions into an overall judgment. For instance, college admissions 
processes that depend on people sizing up candidates are markedly less pre- 
dictive of academic success than adding up scores for the multiple criteria, 
even when some of the individual scores are themselves subjective judgments 
(Dawes 1982). Human judges lean too much toward seeing each case as indi- 
vidual and foregrounding this or that attractive or unattractive feature. 
Unfortunately this makes the final judgments worse, a trend difficult to con- 
vince people of because we are enormously attracted to the idea that nuanced 
holistic judgment always wins over adding up numbers (Galotti 2002, ch. 5). 


Summing Up the Challenges of Decision Making 


Decisions often involve a range of options not apparent at first, as well as 
consequences of character, magnitude, and likelihood that are easily missed 
or mistaken. Quantitative and narrative styles of representation help us to 
manage the complexity, but they are useless if we do not seek out that com- 
plexiy. Instead, intuitive impressions rule. Often they are not well grounded 
in rich experience and personal sensibility. Often, misleading heuristics and 
biases are in the mix. Thus, thoughtful, careful decision makers must attend 
to and watch out for a number of factors. 

However, relatively intuitive approaches sometimes yield good service, 
and more elaborate analyses sometimes yield no better choices and occasion- 
ally worse ones by disrupting sound intuitions. The trade-offs between rela- 
tively more analytic and intuitive approaches as well as between quantitative 
and narrative approaches lead to the question: What in the end is good deci- 
sion making? If we want to educate people to make better choices, what prac- 
tices should we be cultivating? 


GOOD DECISION MAKING 


The quality of decision making can be evaluated in two ways: by examining 
how well particular choices work out and by examining the processes leading 
up to those choices. The first approach 1s, in a sense, the final word on the 
matter. However, systematically applying it is hard. 
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For one problem, decisions often fall short or indeed succeed because of 
unforeseen and not reasonably foreseeable events. Moreover, we often have 
no sure way of knowing what would have happened had we chosen different- 
ly. Ifa year ago we bought stock A rather than stock B, today we can look up 
how stock B did; but if we took job A rather than job B, who’s to say how job 
B would have gone? 

Furthermore, another troublesome cognitive illusion, hindsight bias, 
makes it difficult to judge what choice should have been made earlier know- 
ing only what one knew at the time. People drastically overestimate the 
before-the-fact likelihood of events they know have actually occurred, and 
they also tend to view outcomes as unsurprising —“everyone knows that!” In 
one study, students read accounts of obscure historical events with different 
outcomes reported for different students and one group of students receiving 
no information about outcomes (Fischhoff and Beyth 1975). Asked to esti- 
mate the likelihood of various outcomes before the events, students who read 
that a particular outcome had actually occurred assigned it a much greater 
probability than the no-outcome group. In another study, Teigen (1986) had 
one group of students rate the “truth” of a proverb (e.g., “absence makes the 
heart grow fonder”) while another group rated their “truth” of its opposite 
(e.g., “out of sight, out of mind”). Each proverb and its opposite generally 
earned “true” ratings. 

In summary, defining good decision making by whether the particular 
choice actually works out well is a problematic approach. We do better to 
examine the character of the decision-making process itself. 


The Limits of Utility Models 


As mentioned earlier, another way of characterizing good decision making 
involves the idea of expected utility. The utility strategy calls for constructing 
a tabular representation rating each option on each criterion, with the criteria 
weighted for their importance. If all this is done right, the decision recom- 
mended by the model can be shown to maximize utility for the decision 
maker (e.g., Baron 1988). Accordingly, the utility model might be a tempting 
normative standard for good decision making, providing a rigorous founda- 
tion for educational efforts. 

However, the utility model does not offer the universal answer it may 
seem to at first. The model leaves out its sometimes high cost-in-effort and 
also assumes the decision maker’s capability and available time. Moreover, the 
conclusion that utility will be maximized requires several background 
assumptions. For instance, all relevant options and criteria need to be in play, 
yet decision making often suffers from neglecting a less obvious option or 
criterion. Decision makers’ ratings have to reflect their true goals and values, 
which sometimes are not consciously accessible. 

Also the criteria need to add to the overall utility independently of one 
another rather than interacting. However, one characteristic commonly 
enhances or undermines the value of another. For instance, in a stereo system 
the contributions of speaker quality and amplifier quality are not indepen- 
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dent: Higher-quality speakers contribute less without a good amplifier to 
drive them. 

Such reservations do not challenge the considerable utility of the utility 
model, which often proves valuable in situations with several criteria and 
many tradeoffs across different options. This can easily happen for competing 
purchase decisions or job opportunities with different advantages and disad- 
vantages. The quantitative/tabular approach provides a way to represent and 
review the situation, even when the ideal conditions for its application are 
not met. The issue is not whether the utility model is a good tool in the 
toolkit—it is—but whether it provides a universal normative model for 
good decision making—it doesn’t. 


GOOD DECISION MAKING AS A DOUBLE BALANCING ACT 


If the utility model does not offer the perfect account of a good decision- 
making process, what does? Perhaps no one template will suffice. Effective 
strategy requires sizing up the situation; it requires “deciding how to decide.” 
Good decision making is thus a situation-responsive double balancing act of 
the intuitive and analytic, narrative and quantitative methods suited to the 
circumstances. 

That is, good decision making is an adaptive response to the particulars 
(Payne et al. 1993), which might help to explain the finding from Galotti and 
colleagues (2006) that students’ self-reported decision-making styles (e.g., 
more intuitive versus analytic) were not predictive of the number of options 
or attributes they considered in evaluating college majors. The demands of 
the specific decision may influence what people do more than their stylistic 
predilections. This would fit the general social science observation that 
behavior is determined more by situational characteristics than by personal 
characteristics (Ross and Nisbett 1991). 

Deciding how to decide can be an almost reflexive or a highly deliberative 
matter. In any case, it often will lead the decision maker to a middle level of 
investment in decision making that neither risks an extremely intuitive deci- 
sion nor bears the high costs of the most elaborate styles of analysis. For 
example, one line of experimentation asked subjects to consider options such 
as possible apartment rentals, applying a number of criteria to a range of 
alternatives. Faced with many options, people typically will quickly eliminate 
most of them as failing to meet one or another condition they view as neces- 
sary (for several similar examples, see Galotti 2002, ch. 4). A shopper might 
cross off any apartment on a busy street and any apartment over a certain cost 
per month. Such tactics reduce the options to a few inviting closer considera- 
tion. Strictly speaking, the elimination process is a normative mistake unless 
the elimination criteria are truly necessary conditions. The strategy risks miss- 
ing the rare hidden gem; for example, an over-budget apartment so beautiful 
and convenient that one rethinks one’s budget. However, the strategy makes 
the decision process much more manageable. 
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A narrative approach also can bring the decision maker to a middle level 
of investment, one that is more than a quick take but much less than a saga. 
Recall how research on expert decision making reveals clear patterns of narra- 
tive elaboration, such as testing promising solutions with mental simulations 
(Klein 1998; Lipshitz et al. 2001). Other strategies for making the most of 
one’s narratives without letting them become hideously complex include set- 
tling time, such as “sleeping on it” or taking a walk, and repeated rehearsal to 
reappraise a narrative. I remember one person telling me that he talked over 
decisions with his cat. A deliberate counternarrative, such as the defense’s 
response to the prosecution, is another tactic and a good way to test the 
integrity of a narrative. 

Narrative and quantitative modes of representation involve trade-offs that 
affect how well they suit a particular situation. Narratives can deflect needed 
critical examination through the neatness of the stories they tell. Studying 
informal everyday reasoning, Perkins et al. (1991) argued that people taking 
positions on controversial issues often displayed a makes-sense epistemology, 
suspending critical judgment as soon as they arrived at one way of telling the 
story that made sense. Also, narrative forms quickly become unmanageable as 
they incorporate more dimensions and details. Quantitative/tabular forms 
can accommodate many options and criteria readily if tediously just by 
adding rows and columns. 

On the other hand, even simple quantitative forms are hard to track with- 
out the help of pencil and paper or a spreadsheet. The causal-intentional char- 
acter of narrative and counternarrative supports memory. Also, a narrative 
embodies an explanatory account of the promise of an option that might be 
more revealing than toting up scores. 

Some situations clearly lend themselves more to a quantitative or narra- 
tive style. Apartment features like price, number of rooms, and location can 
be evaluated relatively independently of one another for their contribution to 
the overall desirability of an apartment, thus inviting a quantitative style. In 
contrast, to evaluate a poem by scoring it on independent dimensions of 
rhythm, metaphor, allusion, and so on would be odd: too much depends on 
how the various features integrate with one another. 

Finally, mixed modes are often natural. For instance, a narrative style may 
morph into a quantitative style as more and more factors surface for consider- 
ation and demand organization. Alternatively, a quantitative exploration 
might reveal interacting factors that crystallize into a narrative more com- 
pelling than a tally of scores. Deciding how to decide is not just a single 
choice as one tackles a decision but a choice remade as ideas and information 
develop. 

With more intuitive and analytic narrative and quantitative modes of rep- 
resentation available, what are some of the factors that inform a good double 
balance? At least three are worth considering: stakes, knowledge, and personal 
resonance. 
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The Significance of Stakes for Balanced Decision Making 


Stakes refer to the import of a decision for oneself and relevant others. Stakes 
vary enormously, from trivial, such as what flavor of ice cream to order for 
dessert, to substantial, such as whom to marry or whether to accept an entic- 
ing job offer. In the double balancing act of decision making, the basic guide- 
line is simple: High stakes recommend more investment in narrative and/or 
quantitative analysis to improve the quality of the decision, assuming avail- 
able time and information. 

High stakes do not necessarily favor a quantitative style with options, 
attributes, and ratings. The appropriateness of an analytic approach depends 
on the character of the decision. However, high stakes do caution against 
leaving either the quantitative or the narrative mode out of the picture. The 
two offer different perspectives, they do not necessarily yield the same recom- 
mendation, and they can function as checks on each other, with discrepancy 
an invitation to deeper processing. 

Although high stakes recommend significant analysis, significant does not 
necessarily mean maximal. Often more and more investment in thinking 
through a decision encounters sharply diminishing returns. On the narrative 
side, efforts to explore complex stories and counterstories can proliferate end- 
lessly in the absence of key information. One example of this is the construc- 
tion and appraisal of conspiracy theories around the Kennedy assassination. 
Nor do we need the grand scale of national events to doubt whether an elab- 
orate tale with shaky grounds reveals what’s really going on. Everyday institu- 
tional life in corporations and academic institutions displays plenty of tenu- 
ous tale spinning. 

Elaborate quantitative approaches also commonly show diminishing 
returns. In principle, good decision making calls for keeping even seemingly 
weak options on the checklist: in the search for an apartment, an initially 
unlikely prospect might prove competitive because of high scores on features 
neglected in a first impression. However, if the initial culling ofa long list is 
not too aggressive, this is unlikely. . In principle one can improve the quality 
of a decision by carefully weighting criteria or estimating the probabilities of 
various options succeeding— but if one cannot reliably do so, one may just be 
introducing noise into the analysis. Also, criteria are often correlated with 
one another, in which case relative weighting does not matter so much. 
Dawes (1982), studying the use of utility models to predict graduate school 
performance based on admissions criteria, found that different weightings of 
different criteria—say grade point average and GRE scores and certain inter- 
viewer impressions — did not matter because they overlapped a lot. One 
might as well keep it simple and weigh them all equally. 

A final strategic point for deciding how to decide says that the stakes 
might not be quite as high as they seem. For some decisions, how one pur- 
sues the opportunities of a choice once made may be more important than 
which among reasonably promising options one selects. For example, the 
“right” college or graduate program seems as much a matter of what one 
makes of it as which one picks. 
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The Significance of Knowledge for Balanced Decision Making 


Knowledge, experience, and understanding of a situation are tremendously 
important in the balancing act of good decision making. Fundamental to the 
pursuit of sound choices is taking stock of what one knows and critically 
appraising it. For example, in purchase decisions people assume a strong cor- 
relation between price and quality. However, such relationships vary consid- 
erably. In product areas where standards are enforced and prominent brand 
names figure, price differences are likely to reflect what manufacturers spend 
on advertising more than anything else. Buy generic drugs! 

Consulting with others potentially more knowledgeable is a common 
recourse in decision making—but it also requires critical assessment of the 
input. Are these others really more expert than you? Do they have any reason 
to be biased? Would you do better to trust the most expert among them or 
go with the trend of their responses (on “the wisdom of crowds,” see 
Surowiecki 2004)? Individual consultations aside, the age of the Internet 
makes certain kinds of information very accessible but not necessarily very 
reliable. For example, a quick Internet search for the promise of weight- 
reducing drugs reveals a cascade of strong claims in scientific language, but 
the claims are roundly debunked by reputable medical Internet sites. 
Assessing reliability remains a crucial part of the process. 

Besides the search for objective information, there is the knowledge of 
lived experience. Rich experience in a domain makes possible and even rec- 
ommends a somewhat intuitive narrative mode, especially when the stakes 
are low to moderate or time is a factor. Expertise enables the decision maker 
to build quickly an understanding of the situation from which a viable path 
forward emerges. “Expert” need not mean professional expertise but simply 
the sort of experience a serious weekend gardener or experienced commuter 
might attain. Byrnes (1998, 38) notes four sources from which decision mak- 
ers can draw options: memory, analogical reasoning, causal reasoning, and 
advice. The first three benefit from a reasonable measure of experience in the 
area in question. 

Experience can, however, prove entrapping: The better constructed the 
box, the harder to think outside it. So long as the solution is somewhere “in 
the box,” experience should help. However, sometimes the solution lies else- 
where. The history of innovation includes many examples where entrenched 
belief systems stood in the way of novel theories —for example, the theory of 
continental drift and the bacterial theory of ulcers offered much-resisted 
counternarratives to the received narratives of their times. For more personal 
decisions, seriously pursued counternarratives can help people shake a false 
sense that they know the score. If yourre afraid that leaving the formality of 
the workplace for the freedom of retirement will leave you at loose ends, dis- 
organized and with little to fill your days, ask yourself to envision in some 
detail what sort of life you might construct. If you have always felt you would 
be uncomfortable in an urban environment, ask yourself to envision in some 
detail what life might be like in zzs urban setting today. 
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With some choices, we don’t have enough pertinent experience to tell 
ourselves reliable stories. We are rare visitors to many of life’s puzzles. Not 
often do we face major medical or care decisions for aging parents. Gut 
responses are likely to be unreliable. When we are not well-oriented, deliber- 
ately seeking out information and elaboration is especially important, and the 
more detached quantitative style, even if only in the form of a pro-con list, 
provides a way of organizing what information we do have or can gather 
from friends and other sources. This information can then be supplemented 
by whatever sensible stories we can tell ourselves. Also, we can try to remind 
ourselves about the sorts of heuristics and biases discussed earlier, lest they 
lead us to construct superficially persuasive but poorly grounded stories. 

Situations where we are “visitors” shade into even more challenging situa- 
tions where we find ourselves profoundly disoriented, unsure about how to 
characterize the problem at hand. Consider a medical situation where the 
doctors themselves do not know what is going on, or financial anomalies 
whose cause is unclear and could be the result of computer error or fraud or 
sloppiness. In such circumstances, a quick decision might be ill advised. 
Deciding how to decide might best be a matter of deciding not to decide yet! 
If doing so is safe, stall, buying time to build a better understanding. If deci- 
sions of some sort must be made, sometimes they can take the form of 
hedges or choices that can later be reversed. 

Research on expertise warns that we sometimes are not the experts we 
think we are. Overall, people’s judgments in college admission processes 
actually are not as predictive of academic success for admittees as simply 
adding up scores (Dawes 1982). Extensive experience with an activity does 
not always add up to discernment. Consider how many people never get 
good at sports or games they regularly play. Or consider how some people 
have a pattern of troubled relationships, falling into the same pitfalls over and 
over. Investigators have singled out some of the characteristics that help us 
build reliable judgment from extended experience: recurrent rather than 
unique situations, a greater focus on things than on human behavior, feed- 
back available regarding success rather than no feedback, stable versus 
dynamic situations, and the like (Shanteau 1992). 


The Significance of Personal Resonance for Balanced Decision Making 


For some kinds of decisions more than others, one’s personal intuitions 
about the situation have an intrinsic relationship to the desired outcomes. 
Compare investing $10,000 in the stock market with spending $10,000 on a 
painting for your living room. If you are an experienced investor, you might 
have a feel for the merits of the stock, but the ultimate aim is financial return. 
In contrast, the ultimate aim of purchasing the painting is to acquire an 
object that you will still enjoy looking at many years later. Your feelings about 
the painting today are a sample of what you ultimately want out of it. 

An intrinsic relationship between one’s intuitive feelings and the matter at 
hand informs many life decisions, for instance regarding the relationships one 
sustains, the place where one lives, or the profession one pursues. One can 
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compose a narrative or quantitative analysis for any of these high-stakes deci- 
sions, but to make such choices without asking whether one /ked the other 
person, the dwelling, or the profession would be odd. In such cases, the intu- 
itive voice of personal resonance (or dissonance) is a tremendously important 
data point, often approaching a necessary condition. 

When personal resonance has special bearing, good narrative elaboration 
may serve the decision process better than quantitative/tabular elaboration. 
Consider how poetry or prose can capture essential personal qualities. In 
contrast, decisions regarding stock investments, insurance policies, and wash- 
ing machines have a kind of neutrality that invites treating them largely in 
terms of objective advantages and disadvantages. 


Balanced Decision Making in Summary 


Good decision making can be seen as a double balancing act. The decision 
maker decides how to decide, adopting a more intuitive or analytic approach 
in a more narrative or quantitative or mixed style suited to stakes, knowledge, 
and personal resonance. 

e High-stakes decisions recommend a more analytic approach with both 
narrative and quantitative/tabular representations to crosscheck one 
another but with caution about diminishing returns for extended elabora- 
tion. 

e Extensive knowledge and experience enable more effective use of intu- 
ition and narrative modes of representation—but with awareness that the 
situation may require “thinking out of the box” in ways that challenge 
one’s knowledge and experience. 

e Personal resonance gives a special priority to personal intuitive reasons — 
not because they offer a more accurate read on an objective reality but 
because they represent personal affinities or aversions that are likely to 
continue to color one’s experience of the path taken. 

e Finally, even a relatively intuitive decision of the moment deserves due 
analytic consideration of commonly overlooked matters such as opportu- 
nity costs, cognitive illusions (e.g., loss aversion), and biasing influences 
(e.g., social pressure). 

Although such principles and their refinements offer no calculus of nor- 
mative correctness, they do show how the thoughtful decision maker might 
take stock and choose a smart approach. However, as the saying goes, advice 
is cheap. What does it take to put good decision-making practices into action 
in learners’ everyday lives? 
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Fostering better decision making might seem simply a matter of informing 
learners about how to play the game. Here are the good moves and a little 
practice, now go forth and decide better! However, information and exercise 
alone are not likely to do the job. 
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First, good decision making, as with any everyday kind of thinking, is 
highly dispositional. Appropriate knowledge and skills need the company of 
alertness to occasions and readiness to engage them seriously and thoughtful- 
ly. Several lines of research argue that sophisticated thinking depends not just 
on abilities but dispositions, with people commonly performing below 
capacity for lack of commitment or sensitivity to occasion (e.g., Dweck 2000; 
Perkins and Ritchhart 2004; Perkins et al. 2000; Stanovich 1999). Individuals 
vary considerably in their openness or aversion to the sorts of ambiguities 
and complexities that emerge in working through a difficult choice, as 
gauged by such dispositional indices as need for cognitive closure (Kruglanski 
and Webster 1996) or need for cognition (Cacioppo et al. 1996). For instance, 
tricky, emotionally loaded decisions can prompt costly procrastination during 
which the problem becomes greater— especially when the decision maker 
postpones the decision without using the additional time to gather informa- 
tion or counsel. 

Besides appropriate dispositions, good decision making calls for self-regu- 
lation (Byrnes 1998). Effective decision makers decide how to decide in par- 
ticular situations, monitor their progress, perhaps revise their strategies in 
midstream, and take stock afterward to improve future practice. Accordingly, 
interventions should foster relevant skills and knowledge, cultivate positive 
dispositions, and develop metacognitive self-regulation of decision-making 
practices. 

Byrnes (1998) among others has pointed out two fundamentally different 
approaches to cultivating better decision making. One targets high-priority 
areas such as adolescent sexuality, conflict and violence, drug and alcohol use, 
or parenting practices. The other aims at teaching general decision-making 
skulls and dispositions. 


Teaching Decision Making in High-Priority Areas 

Adolescence offers an attractive zone for interventions because of patterns of 
adolescent risk-taking. Byrnes cautions that the surge in risk-taking does not 
appear to reflect anything about the teenage mind as such but rather the 
reduction of parental controls during that period. Indeed, adults show simi- 
lar patterns of risk-taking. However, adolescents as a group still in school 
provide a convenient and important intervention population. 

Programs focusing on adolescent sexuality and risk provide a class of well- 
researched examples of the targeted approach. Reyna et al. (2005) provide a 
convenient review in the context of discussing a particular developmental 
theory of decision making. “Just-say-no” programs that push abstinence 
appear not to work well, their impact on sexual restraint and taking precau- 
tions minimal. Moreover, abstinence programs can hardly be considered 
efforts to foster decision making. Instead, they promote a particular decision. 
More effective programs engage the student in understanding the risks of 
sexual activities and the ways of protecting oneself and one’s partners. Several 
such programs have been shown to produce near-term effects on rates of 
abstinence and use of protection. 


DECISION MAKING AND ITS DEVELOPMENT 


Often the impact fades in a few months. However, the authors particular- 
ly like a program called Reducing the Risk: Building the Skills to Prevent 
Pregnancy, STD, and HIV (Barth 1996) for which studies suggest somewhat 
longer-term effects. Reyna et al. (2005) foreground some critical features of 
the program’s pedagogy. Students learn what the dangers are, what can be 
done about them, and their own capabilities to act effectively. Role-playing 
and other techniques help develop in-the-moment dispositions and capabili- 
ties. “Social inoculation” equips students to deal effectively with social pres- 
sure. All this unfolds over a considerable period of time. 

What makes such interventions effective? They speak more to intuitive 
decision making in a particular context than to analytic decision making in 
general. They equip learners with a range of basic understandings that offer 
clear immediate implications for sexual caution. Reyna et al. (2005) character- 
ize this as a matter of accumulating and using appropriate “gists,” relatively 
simple ways of understanding situations that allow ready inference of appro- 
priate courses of action. 

For another example, the Resolving Conflict Creatively Program (RCCP) 
focuses on violence-prevention with elementary school children (Aber et al. 
1998). Some fifty lessons foster attitudes and skills around deciding how to 
handle conflict situations, including building cultural awareness and sensitivi- 
ty to interpersonal and intergroup relationships. The methods include exam- 
ples, role playing, discussion, and reflection. 

Propensity to violence considerably increases during the elementary 
school years. Systematic research on RCCP implementations suggests that 
the program dampens this trend for many learners while not reversing it. 
Results were measured in multiple ways, with hostile attributions to hypo- 
thetical scenarios, response choices to hypothetical scenarios, reported 
aggressive fantasies, depression, and actual conduct problems. Results varied 
depending on gender, age, depth of implementation, and other factors (Aber 
et al. 1998; Aber et al. 2003). 

RCCP and similar programs typically emphasize narrative approaches to 
decision making. Less commonly used are coolly analytical tools such as pro- 
con lists or tables of options crossed with dimensions of evaluation. Reyna et 
al. (2005) argue that in high-risk areas such as sexual behavior efforts to rea- 
son through complex trade-offs of risks and gains can be counterproductive. 
Mote insightful gists, broad understandings that strongly recommend cau- 
tionary practices, afford better management of one’s behavior. The learning 
process in these programs develops practical and prudent expertise and uses 
role-playing rehearsals of highly charged situations to address not only 
knowledge and skill but dispositions and self-regulation. In doing so, such 
programs appropriately treat decisions around sexuality and violence as high 
stakes (e.g. the risk of venereal disease or injury), requiring knowledge and 
experience (built up in part through gists and role playing), and reflecting 
positive and negative personal resonance (sexual attractions or surges of tem- 
per addressed through discussion and role playing). 
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Targeted instruction in decision making in high-priority areas can work, 
even in areas as sensitive and gland-driven as adolescent sexuality and vio- 
lence. But the strength of such programs is also their weakness: By intent and 
design they target particular areas rather than general decision-making capa- 
bilities and dispositions. Many of the important decisions people face are not 
in chronic, high-risk areas but are encountered only now and then. 
Developing a full-scale program of instruction for every type of decision 
would not make sense. 


Teaching Decision Making as a General Craft 


Byrnes (1998) reviews three programs designed to cultivate the general craft 
of decision making, drawing on an earlier review of efforts to teach adoles- 
cents decision making by Beyth-Marom et al.(1991). The GOFER program’s 
emphasis is clear from its acronym: Goals clarification, Option generation, 
Fact-finding, consideration of Effects, and Review (Mann et al. 1988). The 
Personal Decision Making program emphasizes a decision-making process of 
five steps: identifying alternatives, formulating criteria, applying the criteria 
to alternatives, summarizing the results, and self-evaluation (Ross 1981). The 
Odyssey program to teach thinking is an extensive intervention of one hun- 
dred lessons, including a ten-lesson unit on decision making, with attention 
to anticipating outcomes, acquiring and assessing information, articulating 
preferences, and applying weights to dimensions to manage complex deci- 
sions (Herrnstein et al. 1986). 

All three programs address adolescents. All consist of a few lessons on 
decision making to be taught in a school context. All emphasize application 
of multiple attributes to options in order to score them. All operate as inde- 
pendent mini-courses within the curriculum. All include the teaching of spe- 
cific strategies or procedures toward understanding the decision situation, 
identifying options, examining consequences, and evaluating options in 
terms of their consequences. They seek to help learners develop the basic 
skills and dispositions of an analytic, quantitative style of decision making. 

Some interventions with a similar strategic emphasis engage the rhythm 
of schooling in a different way. The literature on the thinking-skills move- 
ment makes a distinction between stand-alone and infused interventions 
(e.g., Swartz and Perkins 1989; Perkins 199s, ch. 8). Whereas stand-alone 
interventions constitute courses or mini-courses of their own, infused inter- 
ventions combine the cultivation of thinking with learning in the disciplines. 

Robert Swartz and his colleagues have worked on infused models of the 
teaching of thinking for many years (Swartz and Parks 1994; Swartz et al. 
2007). Their approach includes considerable attention to decision making, 
fostering attention to options, consequences, costs, and related matters. 
Graphic organizers often are used to help learners express and order their 
thinking around content-related issues. In one typical application, students 
take on the role of Harry Truman pondering whether to drop atomic bombs 
on Hiroshima and Nagasaki (Swartz et al. 1998; for a brief account and 
graphic organizer, see Swartz 2000). The students are allowed to reason only 
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with information available up to 1945. Intervention at early grades is possible 
too. A sample lesson for first graders uses Horton Hatches the Egg by Dr. Seuss 
and a simplified decision-making strategy— What could we do? What would 
happen if we did those things? What’s the best thing to do? 

The designers of infused programs would not expect occasional decision- 
making activities in history or reading to make learners better decision mak- 
ers. Rather, they encourage frequent and consistent use of such practices 
across a variety of disciplines, aiming both to deepen disciplinary learning 
and instill decision-making practices that will spill over to other disciplines 
and to learners’ everyday lives. 

How confident can we be that such initiatives, stand-alone or infused, 
have the desired impact? A prior question concerns the entire enterprise of 
the teaching of general thinking skills and dispositions. This issue has been 
controversial (Anderson et al. 1996; Brown et al. 1989; Perkins and Salomon 
1987, 1989). Some have argued, using concepts like situated learning (e.g., 
Lave and Wenger 1991), that enhancing cognitive capabilities in a general 
sense is unlikely. Effective initiatives need a contextual focus, as with the pro- 
grams on adolescent sexuality and violence. For a while, the debate took a 
polarized “can’t be done / can be done” form. My sense is that a more moder- 
ate position now prevails. In any case, I have argued elsewhere that the litera- 
ture now offers a number of clear empirically grounded cases of enhancing 
various general cognitive skills (Grotzer and Perkins 2000; Perkins 1995; 
Ritchhart and Perkins 2005). 

However, decision making per se has not been the focus of the best 
research. Byrnes (1998) observes that the evaluations of the GOFER, 
Personal Decision Making, and Odyssey programs all included some positive 
results— but on measures that emphasized students’ knowledge of decision 
making. Researchers made no posttest effort to inventory students’ decision- 
making behavior in real-world contexts or to engage them in simulated deci- 
sion-making processes. 

One can hope that the students’ gains in knowledge and understanding 
would translate into shifts in practical behavior, but Byrnes warns that the 
pedagogical approaches taken seem somewhat didactic. Ideas about transfer 
of learning point to features that should enhance impact: metacognitive 
reflection; intensive simulated rehearsal, such as that which is characteristic of 
targeted approaches; and direct encouragement to apply the ideas widely, 
such as log-keeping activities around everyday decision making (e.g., Perkins 
and Salomon 1989; Salomon and Perkins 1989). 

Many examples of infused approaches exist, but little research has been 
done in this area because embedding thinking strategies in disciplinary learn- 
ing introduces many hard-to-control variables. Two kinds of outcomes — 
enhancement of disciplinary understanding and thinking—are relevant, and 
even without controlled studies we can reasonably claim that the extensive 
use of infused decision making or other thinking practices almost certainly 
fosters disciplinary understanding. Extensive research has demonstrated that 
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thoughtful elaborative processing of content builds understanding (e.g., 
Cohen et al. 1993; Meyer and Land 2006; Wiske 1998). 

On the other hand, the impact of infused approaches on students’ general 
decision-making practices lacks rigorous evidence one way or the other. 
Informal reports of deeper writing about decision situations are encouraging, 
as are frequent anecdotes from students about how they applied one or 
another technique. However, these do not offer direct evidence of lasting 
general changes in decision making. Although inconvenient to investigate, 
this is an area that would be worth the effort. 

To summarize, pedagogical approach is important. Didactic styles are 
likely to be less effective than approaches that involve metacognitive reflec- 
tion and simulations. Fairly frequent activities over a considerable period of 
time are likely to have greater impact than a brief unit. Approaches that focus 
on single disciplines or topics are likely to be less effective than approaches 
that deliberately range across disciplines and invite learners to make connec- 
tions to their out-of-school lives, fostering transfer of learning. 

Also, typical interventions such as GOFER lean toward the quantita- 
tive/tabular style of analysis, with options and attributes and ratings or check- 
lists. They also neglect the importance of regulating decision making accord- 
ing to the critical dimensions of stakes, knowledge, and personal resonance, 
generally assuming medium to high stakes and making available necessary 
knowledge: not much attention 1s paid to sizing up the circumstances and 
deciding how to decide. The programs also often do not directly deal with 
typical heuristics and biases, including social influences like bandwagon 
effects. Accordingly, we might look toward interventions that take a more 
balanced decision-making approach, with learners encouraged to develop 
positive dispositions and self-regulation regarding which approach or what 
mixture applies when. 


DECISION MAKING AND CIVIC ENGAGEMENT 


Better decision making has not just personal but social importance. The par- 
ticipative character of democracies calls for informed citizens who thought- 
fully consider matters of policy and justice, adding to the collective wisdom 
by voting, campaigning, protesting, and engaging in informal discourse. One 
of the principal goals of education is generally taken to be preparation for 
such roles. 

Thus, however schools address decision making in general, they would 
do well to engage the peculiar dilemmas of personal decision making for the 
public good. Imagine, for instance, that civic engagement was approached 
much the same way as programs on adolescent sexuality and violence, with 
assiduous attention to naive beliefs, social inoculation against misleading 
pressures, simulation activities to foster behavioral patterns of thoughtful 
civic participation resistant to undue influence, and so on. 

Interventions with such a clear social agenda should include ample atten- 
tion to the contribution of social interactions to decision making. One baleful 
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influence is groupthink (Janis 1972), a phenomenon that runs all the way from 
PTA meetings to national politics. Groupthink is the tendency for group 
members to influence one another in ways that lead to an artificial and mis- 
leading consensus. It’s a matter of following the crowd, deferring to influen- 
tial figures, and keeping contrary thoughts to oneself when a trend gathers 
momentum. 

Quite the opposite phenomenon is documented by James Surowiecki 
(2004) in his recent The Wisdom of Crowds. Surowiecki reviews numerous 
research results and natural circumstances demonstrating that the common 
trend across a number of individuals often yields remarkably good judgments 
and predictions — better than most of the individuals involved, even the 
experts. Cases range from estimating the number of marbles in a jar to pre- 
dicting elections and economic trends. 

The wisdom of crowds is good news for the potential of democratic soci- 
eties and free markets. But it comes with a seeming paradox: How can we 
have both groupthink and the wisdom of crowds? The answer is that they 
reflect different circumstances. Groupthink thrives on close interactive rela- 
tionships between the people involved, so that opinions can snowball. In 
contrast, the wise crowd generally involves people not in close touch, with 
varied sources of information, and reaching independent judgments. 

Although students certainly could prepare better for decision making in 
civic contexts, the challenge goes well beyond individual commitment and 
capability. Thoughtful independent judgment is not so well served by typical 
political processes and institutions. Even reasonably committed members of 
the polity cannot find it easy to function in an ideal way. Consider the inordi- 
nate burden of understanding complex public issues. No matter whether the 
theme is immigration, tax laws, military commitments, or healthcare, the 
media are replete with debate from power figures and interest groups. Every 
statement has a counterstatement. Misinformation is commonplace, as regis- 
tered by watchdog groups that fact-check political speeches. Statistics are 
used to lead and mislead. The principal voices strive for simplistic compelling 
stories that will win allegiance. Sources written in the neutral manner of rea- 
soned opinion are often deeply biased. 

Now consider the challenge from the perspective of balanced decision 
making and the three factors: stakes, knowledge, and personal resonance. In 
civic engagement, stakes often have the unfortunate configuration of low 
personal, high public significance. Many individuals correctly judge that 
large-scale policy or leadership decisions will not much affect their personal 
lives. Even when the personal impact could be great, they note that their 
opinion or vote is a drop in the bucket—their expected return on investment 
of effort is very low. As to knowledge, most people cannot be deeply knowl- 
edgeable about the welter of issues in play, and efforts to acquire some basic 
and reliable grounding are frustrated by the adversarial and often manipula- 
tive character of political PR. As to personal resonance, the natural turn of 
public controversy is to personalize everything with broad appeals to identity 
and emotion—even issues that would better be considered in detached ana- 
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lytic ways. In summary, the typical pattern of public discourse around major 
issues and offices is singularly unsupportive of, indeed in many ways actively 
undermining of, thoughtful citizenly engagement. 

To develop citizenly skills and dispositions while neglecting large-scale 
patterns that undermine thoughtful engagement would be shortsighted. To 
get results, we need to attend to both. One can envision institutions that help 
rather than hinder civic engagement by clarifying stakes, providing reliable 
knowledge, and handling personal resonance in an honest way. Imagine, for 
instance, that major newspapers, regardless of their political leanings, rou- 
tinely made available fact checks of political statements, with lead-ins always 
on the front page. Imagine that Google provided prominent front-page links 
to such analyses for every user who did not explicitly opt out. Imagine that 
the media took a cue from the way technology products are often analyzed in 
reviews, organizing summaries of issues with tables itemizing factors and 
briefly characterizing features. Occasionally one sees this, but it could be 
commonplace. Imagine that important and orienting factual information— 
not information under significant debate—were widely and actively “adver- 
tised” to produce better public calibration around important issues (because 
the reality is that most people remain seriously underinformed even when 
critical information is widely accessible). 

Ideas such as these bring with them many challenges. The point is not 
that any one of these ideas is a magic bullet but that educating for more 
thoughtful civic engagement makes little sense if the social institutions that 
undermine thoughtful civic engagement continue to be disregarded. 


THE FUTURE OF DECISION MAKING 


Just as the role of decision making in individual and collective endeavor is 
enormous, so is the research on decision making extensive. Many prominent 
themes are hardly touched here, including moral aspects of decision making, 
the significance of time pressure, the complexity of rapidly changing situa- 
tions, the detection of important decision points handled by default but mer- 
iting reconsideration, the role of will and intentions in following through on 
in-principle decisions (the “New Year’s Resolutions” problem), goal revision 
in contrast with evaluating options relative to current goals, decision avoid- 
ance, regret avoidance as a driver of decision making, decision-making styles, 
and the intricacies and pitfalls of group decision making. 

Although speaking to all these motifs, the literature also displays signifi- 
cant blind spots. Galotti (1995) notes that relatively little research has tracked 
real-life decision making in nonexperts, her studies of college choice and col- 
lege major choice being an important exception. We know little about which 
heuristics and biases investigated in the laboratory come up the most in real- 
istic circumstances. Evidence positive or negative of the impact of interven- 
tions on actual everyday behavior is sparse. 

These limits acknowledged, the present exploration advocates for more 
attention to be paid to the following areas of neglect: 
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1. Beyond analytical to well-grounded intuitive decision making. Much of the 
literature on decision making and most educational interventions targeting 
general decision making treat it as ideally an analytical enterprise. However, 
research tells us that in certain contexts well-grounded intuitive thinking 
serves decision making better. Accordingly, what has here been characterized 
as balanced decision making, with the right balance sought situation by situa- 
tion as the person decides how to decide, would serve personal and public 
interests better than a doggedly analytic approach. 

2. Beyond quantitative/tabular to narrative styles of decision analysis. Ap- 
proaches to cultivating general decision-making capabilities tend to fore- 
ground quantitative analysis— options, attributes, ratings, scores, or junior 
versions of the same, such as pro-con lists and tallies. This is often a powerful 
mode of analysis, but research on decision making suggests that narrative 
analysis is potent as well. The particular character of the decision can lend 
itself to one or the other or to a blend. 

3. Both high-priority targets and general decision making. While some people 
hope that the cultivation of general decision-making skills and dispositions 
will serve well enough, high-priority persistent trouble spots such as adoles- 
cent sexuality, drug use, violence, and civic participation almost certainly 
require targeted treatment. Good practice depends too much on context-spe- 
cific knowledge encoded in ways that make it fluently available in the 
moment, and on managing tricky aspects of personal resonance, for anyone 
to expect that general decision-making skills and dispositions will powerfully 
inform such areas. 

However, personal and public life include a myriad of medium- to high- 
stakes decisions that occur only now and then. One cannot prepare young 
learners for hundreds of specific decisions case-by-case. Some researchers 
have claimed that general cognitive skills cannot be significantly enhanced, 
but plenty of evidence exists to the contrary. Therefore, general decision-mak- 
ing skills and dispositions are important too. 

4. Beyond improving people to improving institutions. Improving citizenly 
decision making tends to be viewed as solely a matter of improving citizens — 
better people for a better democracy. However, the quality of decision making 
greatly depends on contextual support as well as the skills and dispositions of 
participants. In many ways, current public institutions and patterns of dis- 
course undermine rather than sustain good decision making. This invites 
attention to social innovations that would make thoughtful civic participation 
more natural and responsible—and the same applies to innovations on a small- 
er scale, as within corporate, government, and educational institutions. 

My recommendation is emphatically expansive: Much more can be done 
than is being done. We know a lot today about how decision making might 
go better, and this knowledge provides a platform on which to build. 
Moreover, the best of current programs to improve decision making, both 
targeted and general, seem likely to have some beneficial effects. 

However, comprehensive efforts to enhance decision making should 
include more attention to intuitive and narrative approaches. Efforts to 
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improve general decision making should not be seen as substitutes for 
addressing high-priority targets and vice versa. Making people better decision 
makers should figure as part of a larger agenda of developing social settings 
to be more supportive of thoughtful decision making. 

As to research, we do not know enough about the impact of interven- 
tions, particularly those focused on improving decision making generally. 
Nor do we have good maps of the variety of decisions people face in their 
lives and the pitfalls that cause the most trouble outside of laboratory studies 
and simulations. Such research is challenging because it requires finding ways 
to track people’s decision making “in the wild,” but it is important enough to 
pursue with more vigor. 

In present patterns of schooling as in our society at large, the improve- 
ment of everyday decision making gets meager attention. Yes, one can point 
to the occasional course or other intervention, but the reality is that most 
people spend little learning time on decision making as such. Few areas of 
skull and character promise so much payout and receive so little pay-in. 
Whether to invest in fostering everyday decision making is a decision also. 
These pages are something of an argument for its importance and priority. 
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Can Reasoning 
Be Taught? 


RICHARD E. NISBETT 


Can people be taught to use abstract inference rules, such as the rules of logic, 
to reason about events in everyday life? 

For 2,500 years the Western world believed that the answer to this was 
yes. Plato said “those who have a natural talent for calculation are generally 
quick at every other kind of knowledge; and even the dull, if they have had 
an arithmetical training . . . become much quicker than they would otherwise 
have been . . ” and “We must endeavor to persuade those who are to be the 
principal men of our state to go and learn arithmetic” (Plato 1875, 785). 

The Romans agreed with their Greek predecessors, adding the study of 
grammar to the curriculum. The medieval scholastics were not ones to doubt 
the wisdom of the ancients and added to the curriculum the study of logic, 
especially sylogisms. The humanists of the Renaissance, even more in thrall 
to the ancients, added the study of Latin and Greek, and the curriculum was 
set for the next 400 years, culminating in the English public (sic) school sys- 
tem of the nineteenth century. One educator proclaimed the utility of Latin 


for teaching people how to think: 


My claim for Latin, as an Englishman and a... teacher is simply that it 
would be impossible to devise for English boys a better teaching instru- 
ment. .. . The acquisition of a language is educationally of no impor- 
tance; what is important 1s the process of acquiring it... . The one great 
merit of Latin as a teaching instrument is its tremendous difficulty. 


But the contention that the learning of abstract rule systems has any effect 
on people’s ability to reason about everyday life problems was one of the first 
ideas to be attacked by the new discipline of psychology at the turn of the 
twentieth century. William James ridiculed the idea that the mind had mus- 
cles that could be exercised by arithmetic or Latin. The learning theorists of 
the 1920s and 1930s provided a theoretical rationale for rejecting the idea of 
highly general rules. Behavior and thought consisted of responses to concrete 
stimuli, and what was learned was a limited stimulus-response link. “(T)he 
amount of general influence from special training (is) much less than com- 
mon opinion supposes” (Thorndike 1906, 246). Even the early cognitive 
scientists of the 1960s and 1970s rejected the view that reasoning was much 
influenced by general rules. Alan Newell declared “the modern position is 
that learned problem-solving skills are, in general, idiosyncratic to the task” 
(Newell 1980, 178). 
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An exception to the anti-inferential rules position of many twentieth-cen- 
tury psychologists was Jean Piaget, who held that people do have abstract 
inferential rules, including those corresponding to the rules of propositional 
logic, as well as particular cognitive schemas, including schemas for propor- 
tionality, probability, and the mechanical equilibrium principle of action-reac- 
tion (e.g., Inhelder and Piaget 1958; Piaget and Inhelder 1951/1975). However, 
Piaget was as insistent as other twentieth-century psychologists that such 
rules can’t be taught but only induced from living in the particular world that 
we live in. 

But the antirule, anti-instruction position of much of twentieth-century 
psychology is mistaken. My colleagues and I have shown that people do have 
inferential rules corresponding to a number of inferential rule systems — 
including probability and statistics, methodological principles that are relied 
on by the social scientist, the rules of cost-benefit decision theory, and what 
we call “pragmatic reasoning schemas.” (Larrick et al. 1990) Moreover, these 
rules can be readily taught, some to a significant degree in classroom or labo- 
ratory settings lasting an hour or less. 

Consider the following problem: 


Catherine is a manufacturer’s representative. She likes her job, which takes 
her to cities all over the country. Something of a gourmet, she eats at 
restaurants that are recommended to her. When she has a particularly 
excellent meal at a restaurant, she usually goes for a return visit. But she is 
frequently disappointed. Subsequent meals are rarely as good as the first 


meal. Why do you suppose this is? 


We have posed this sort of question to scores of people having education- 
al levels ranging from college freshman to Ph.D.-level staff at major research 
institutions (Nisbett et al. 1987; Nisbett 1992). From freshmen, we almost 
never get anything other than a causal hypothesis: “Maybe the chefs change 
a lot” or “Maybe her expectations are so high that the reality will disappoint 
her? Such hypotheses are not necessarily wrong, but they miss underlying 
statistical points. From undergraduates who have had a course in statistics, 
we often get answers that reflect an appreciation of the probabilistic nature 
of restaurant meal quality: “Maybe it was just by a chance that she got such a 
good meal the first time,” which is surely right, as far as it goes. From gradu- 
ate students in psychology, who have typically had two or three courses in 
statistics, we usually get a statistical answer, often of high quality, such as 
“There are probably many more restaurants where you can get an excellent 
meal some of the time than there are restaurants where you can get an excel- 
lent meal all of the time. So if she gets an excellent meal it’s probably in a 
restaurant that is only very good on average. Therefore if she gets a truly 
excellent meal then there’s no place to go but down—on average.” From 
Ph.D.s in science we nearly always get a statistical answer, usually of high 
quality. 

Can you teach people statistical principles that can affect their understand- 
ing of everyday life events without having them take hundreds of hours of 
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courses? Yes, we can teach people how to reason about events like Catherine’s 
disappointment in laboratory sessions lasting less than an hour (Fong et al. 
1986). We can teach rules like the law of large numbers in purely abstract fash- 
ion by talking about urns with balls of different colors; defining the concepts 
of population parameter, sample parameter, and sample size; and pointing 
out that larger samples will on average reflect population parameters better 
than smaller samples. We can also teach by presenting people with a number 
of concrete problems in everyday life that require the law of large numbers 
for solutions. For example, we can ask people to think about the following 
sort of problem: 


David is a high school senior choosing between two colleges. He has 
friends at both colleges. His friends at College A like it a lot on social and 
academic grounds. His friends at College B are not so satisfied, being 
generally unenthusiastic about the college. He visits both of the colleges 
for a day and meets some students at A who are not very interesting and a 
professor who gives him a curt brush-off. He meets several students at 
College B who are lively and intelligent and a couple of professors take a 
personal interest in him. Which college do you think he should go to? 


Most people who are uninstructed in statistics think David should go to 
the place he likes, not the place his friends like. But you can massage people’s 
probabilistic intuitions by saying, in effect, “We can think of David’s impres- 
sions of each campus as a sample parameter. But David’s sample is very small 
and could well be misleading. His friends have a much larger sample of the 
colleges, and we would expect that their sample parameters are closer to the 
population parameter of college satisfaction for people like David. So David 
should probably go with what his friends think-” 

Teaching people the rule in the abstract, using arbitrary events like balls in 
an urn, and teaching people the rule using concrete examples like the college- 
choice problem are both effective in getting people to apply statistical solu- 
tions to problems that require them. The two together are even more effective. 
Moreover, the effects last over a period of at least several weeks, and for the 
concretely trained subjects an influence is found for problems far in surface 
content from the problems they were trained on (Fong and Nisbett 1991). 

We find similar benefits for teaching people principles of behavioral 
science methodology. Think about the following problem: 


The promoters of a local Lose Weight Now! organization have claimed 
that, on the average, their members lose ten pounds during their first 
three months of attending meetings. To test this claim, a public health 
nurse kept records of weight lost by every new member who joined the 
Lose Weight Now! branch during 2006-2007. Out of 138 people who 
started to attend meetings, 81 kept attending for at least three months, 
and, indeed, the average amount of weight lost by these people was 9.7 
pounds. Does this study establish that the Lose Weight Now! program is 
effective in helping people to lose weight? 
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The flaw in any conclusion that the program is effective is the “self-selec- 
tion” possibility. That is, the people who stuck with the program may have 
been those who were going to lose weight anyway. Those who didn’t stick 
with the program may have been those who weren’t losing weight and had 
given up. They might even have gained weight. That you’re going to lose 
weight if you enroll in the Lose Weight Now! organization is not clear. 

We find that two years of graduate training in psychology greatly increas- 
es the likelihood that people will spot the potential artifact in problems that 
require understanding of the self-selection principle, or the relevance of con- 
trol groups, or consideration of the base rate for a given outcome (Lehman et 
al. 1988). Medical training improves people’s solutions to such problems some- 
what, and training in chemistry and law does nothing whatever (Lehman et 
al. 1988). We have not tried laboratory training sessions to teach such princi- 
ples, but I don’t doubt that this could easily be done, with results that would 
be lasting. 

How about logic? Taking a course in formal logic actually does nothing 
for the ability of undergraduates to reason about everyday life problems that 
require the logic of the conditional (Modus ponens, Modus tollens, etc.; 
Lehman and Nisbett 1990). 

Even two years of graduate education in philosophy does nothing for 
conditional reasoning, although it is effective for some types of syllogistic 
reasoning and for the ability to come up with damaging counterarguments 
to a proposition (Morris and Nisbett 1992). 

I would explain the difference between the teachability of statistical and 
methodological reasoning on the one hand and the difficulty in teaching logi- 
cal reasoning on the other as being due to the differential “gracefulness” of 
increments to the rule system. People already have rudimentary, intuitive ver- 
sions of probabilistic and methodological rules, and when we teach them we 
are improving on rule systems about which they already have some inkling. 
The difficulty in teaching logic may be a matter of alarm to some, but not to 
me. I think the rules of logic fall into two categories—those everybody 
induces by virtue of long practice in the world, such as “or exclusion” (either 
Aor B is the case but not both), and those that are highly artificial, such as 
the more nonintuitive implications of the conditional or most syllogistic 
forms. (Bertrand Russell said about the medieval monks’ development of the 
syllogism that it was as barren intellectually as they themselves were repro- 
ductively.) 

How about cost-benefit rules? Are people capable of making their choices 
in the highly rational fashion required by the formal axioms of choice? Econ- 
omists have gone through three phases on this question. In the first phase, 
economists maintained that all choices are in fact made in accordance with 
those abstract rules. Then Herbert Simon loosened the requirements a bit by 
introducing the concept of “bounded rationality”: given the brevity of life, 
people consider choices only to the extent that they are important and infor- 
mation about the relevant utilities and probabilities are easy to come by 
(Simon 1955). This bounded rationality sounded to most people as if it were 
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rational enough. But then research, especially that by psychologists Daniel 
Kahneman and Amos Tversky, showed that people weren’t even very bound- 
edly rational. They spend as much time shopping for a shirt as for a refrigera- 
tor; they are risk averse in situations where there is a potential gain and risk- 
seeking in situations where there is a potential loss; they are subject to severe 
framing effects (for example, reaching opposite conclusions about the same 
formal problem when encouraged to think about the problem as a possible 
gain than they do when encouraged to think about it as a possible loss); they 
calculate value not with respect to some absolute scale but merely with respect 
to their current state; and they don’t use proper probabilities at all but rather 
something more like “decision weights”; and on the list could go (Tversky & 
Kahneman 1981). 

While not denying any of these pejorative characterizations of the deci- 
sion maker, I have good news. Formal principles of cost-benefit analysis can 
be taught in such a way as to make people more likely to employ them in 
everyday choices (Larrick et al. 1990). Moreover, people are better off when 
they do use those principles.’ 

Consider the following problem: 


Several months ago you bought tickets to a basketball game in a nearby 
city. That game will be played tonight. However, the star of your team is 
not playing, the opposing team is weaker than expected, and snow has 
begun to fall. Should you go to the game or tear up the tickets? 


If you said you should go to the game because it would be uneconomical 
not to consume something you've paid for, yourre not thinking like an econo- 
mist, who has a valuable rule for making such decisions; namely, the “sunk 
cost” rule, which follows from the choice axioms. This rule says that you should 
not consume something that you’ve paid for, unless its value 1s positive at the 
present time. You've already spent the money; it’s sunk. You cart retrieve any 
part of it by suffering through an unpleasant and possibly risky drive to watch 
a game that is likely to be boring. 

Do economists live their lives using such principles? We gave policy prob- 
lems and everyday life-choice problems to University of Michigan economists, 
biologists, and humanities professors (Larrick et al. 1993). They included, in 
addition to sunk cost problems, “opportunity cost” problems, where the trick 
is to recognize that you shouldn’t pursue some course of action when another 
course of action offers a likely lower cost and a likely higher benefit. For exam- 
ple, we gave study participants a sunk cost problem that asked whether they 
agreed with the university’s decision to tear down the old hospital and put up 
a new one, even though the cost of putting up a new one was nearly as high as 


1. The arguments by economists on this latter point are not very persuasive: The cost- 
benefit rules must be beneficial because 1) under the assumptions made by economists they 
can be shown to maximize outcomes and 2) corporations pay decision experts for advice. 
Unfortunately, corporations also pay for handwriting analysts, lie detectors, and motiva- 
tion experts to jump around on stage to unknown effect. 
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the cost of renovating the old one and the old one had been extremely expen- 
sive to build. Economists and everyone else should say that the cost of the old 
hospital is sunk and therefore irrelevant to the present choice. 

Economists do in fact say that, and they are much more likely to apply 
cost-benefit principles in the avowedly normatively correct way to all kinds of 
policy problems and personal choices (e.g., Walk out of a lousy movie or stay 
to the bitter end?) But biologists and humanities professors are not nearly as 
likely to use those principles as economists. And biologists and humanities 
professors are not much more likely to use those principles than their students. 

How about training in economics short of a doctorate? Does a course in 
economics make you more likely to use the cost-benefit rules? No. Under- 
graduates who have had a single course in economics are no more likely to 
use cost-benefit rules in analyzing policy questions or making personal deci- 
sions than are undergraduates who have never had a course in economics 
(Larrick et al. 1993). 

But we can teach college students how to use the sunk cost and opportu- 
nity cost principles in sessions lasting less than an hour (Larrick et al. 1990). 
And the new rules stick around. Two weeks after training them, we call them 
in the guise of an opinion poll and ask questions that don’t look at all like the 
ones on which they were trained. The students use the normatively correct 
rules more than do untrained students. 

But how do I know they’re better off using “normatively correct” rules? 
I’ve already admitted that neither the abstract optimality nor the fact that cor- 
porations hire decision experts provides much reason to believe that the rules 
are correct. 

But our research shows that use of the rules is associated with life out- 
comes that people desire. Faculty members who regularly use cost-benefit 
rules when making choices get higher salaries and bigger raises than do those 
who less regularly use the rules (Larrick et al. 1993). Undergraduates who 
more frequently use the rules have higher GPAs. One might ask whether this 
finding can be attributed to the self-selection principle: Are the students who 
use the rules, or claim to do so, more intelligent than those who do not use 
the rules? Actually, students who have higher grades than their SAT scores (a 
pretty good indicator of intelligence) would predict are more likely to use the 
rules. Being wise about choices helps you to achieve more than you would 
otherwise. 

To date we have studied and found effective ways of teaching a significant 
number of rule systems. We will continue to identify many more inferential 
rules that are pragmatically helpful to people, and we will also find surpris- 
ingly efficient ways to teach those rules. 
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Is Critical Thinking a 
General Talent? 


JEROME KAGAN 


Human beings demand that any activity that exacts a cost in money, effort, 
and energy must have a purpose that is ethically praiseworthy. Formal educa- 
tion is expensive, demands continued perseverance by teachers and pupils, 
and expends psychic and biological energy. Thus, to inquire into its purposes 
is appropriate. 

The ancient Athenians had no difficulty providing an answer to this query. 
The assignment given to Athenian educators was to produce the next genera- 
tion of responsible citizens. This category had moral connotations because its 
critical features —which applied only to male citizens, not to females or 
slaves—were respect for the gods, loyalty to the polis, and perfecting one’s 
talents. The clergy who administered the church schools of Medieval Europe, 
like those in charge of the Islamic classes in Islamabad today, understood that 
their goal was to inculcate an unquestioning religious faith and commitment. 

Late-nineteenth-century Americans, sensing their country was on the 
cusp of expanded wealth and power, recognized that their recent European 
immigrants had to learn to read, write, do sums, and adopt an ethic of self- 
reliance in order to hasten their identification with their new national catego- 
ry. And, remarkably, this curriculum worked well for about four generations 
until history’s muse, bored with the lack of excitement, rearranged the actors, 
scenery, and plot line. Contemporary Americans and Europeans, confronting 
an extraordinary diversity in mores, religion, and country of origin within 
their populations appreciate that these social facts demand an unusual level of 
tolerance for all ethical views and a willingness to suppress the natural temp- 
tation to rely on one’s ethnicity, religion, or family pedigree for reassurance 
of selfs virtue when this stabilizing belief is momentarily threatened. As a 
result, wealth and the enhanced status associated with a professional vocation 
have become primary signs of a person’s dignity. But the combination of a 
technological economy and the failure of public schools serving economically 
compromised urban neighborhoods to prepare their youth for the intellectu- 
al demands of contemporary society have made it more difficult for marginal- 
ized youth to become upwardly mobile. Schools have an important responsi- 
bility in helping to solve this problem. 

One would think that educators would have responded by teaching 
the linguistic, mathematical, and scientific competences the new economy 
required. But their inability to motivate many disadvantaged youth to master 
these domains in order to choose technical vocations was embarrassing. 
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Although the schools bear some of the blame, the families of these children 
also share responsibility for the failure. But because blaming victims for their 
errors has become politically incorrect, the schools took the brunt of the criti- 
cism and sought a goal that seemed to be more attainable, socially relevant, 
and ethically neutral enough to avoid offending the public. 

An enthusiasm for promoting “critical thinking” appeals to some as an 
answer to this social need. Students should be able to evaluate evidence so 
that, as adults, they can arrive at more correct judgments of the truth or 
falsity of popular claims. But Poincare understood that facts do not speak. 
The remote sections of Widener Library contain volumes full of facts, but 
the passageways are remarkably quiet because a mind confronting a corpus 
of facts must have some a priori premises and a body of relevant knowledge 
in order to begin the task of evaluating their truth or falsity. Imagine a person 
unfamiliar with the physics of light who sees a pencil resting in a tall glass 
half-full of water. The evidence this scene presents is perceptually compelling. 
The pencil appears to bend at the point where it penetrates the surface of the 
water. Lacking the principles of physics, the person might conclude that 
water can bend pencils. 

The assumption that an abstract psychological competence called “critical 
thinking” can transcend the body of information being analyzed is as flawed 
as the belief that honesty, empathy, and civility are traits that most individuals 
display across varied settings. Unfortunately, the idea of “critical thinking?’ 
independent of the context in which this function is applied, resembles New- 
ton’s concept of the ether or Wechsler’s concept of IQ. Neither is a phenome- 
non in nature; not a flamingo but a unicorn. 

Consider some examples of highly intelligent adults who enjoyed excel- 
lent educations and community respect because of their ability to evaluate 
the evidence in their restricted domain of talent. Francis Collins, a biologist 
respected by colleagues and the head of the Human Genome Project, con- 
fessed that his evaluation of the scientific data led him to the conclusion that 
God intended the forms and mechanisms that represent the facts of modern 
evolutionary biology; Richard Dawkins, brooding on exactly the same evi- 
dence, came to the opposite conclusion. The icon of critical reasoning, Albert 
Einstein, rejected the new ideas in quantum mechanics because they denied 
determinism; Niels Bohr found the new theories perfectly reasonable. Kurt 
Gédel, author of the incompleteness theorem and a scholar Einstein regarded 
as a better logician and mathematician than he, had his wife taste his meals 
first because he believed that his food might be poisoned. A distinguished panel 
of American intellectuals appointed by the National Academy of Sciences 
published a report, which made newspaper headlines in 2006, declaring that 
their critical evaluation of all the evidence led them to conclude that there 
were no biological differences between the sexes with implications for human 
behavior or cognitive abilities. The majority of biologists, neuroscientists, 
and psychologists who were familiar with the same corpus of evidence were 
surprised by this judgment. In their opinion, the panel’s conclusion was dra- 
matically inconsistent with the facts. These few examples should be sufficient 
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support for the claim that one can teach critical thinking within a specific 
domain of knowledge but cannot teach critical thinking as an abstract talent 
that transcends domains. Could one teach “athletic skill” outside of a particu- 
lar sport or “aesthetic appreciation” independent of the form the artistic 
product assumed? 

Teachers can train children, in the first eight or twelve grades, to evaluate 
the correctness of answers to a set of long-division problems, to decide on 
the amount of food and water a cage of gerbils requires in order to survive, 
or to estimate the amount of raw sewage that if emitted from a factory into 
a river that would kill life-forms. Americans should be able to evaluate news- 
paper reports claiming that vitamin E prolongs life or that the nursing of 
infants has a benevolent effect on their future health. But the correctness of 
these evaluations depends on possession of specific knowledge. If citizens do 
not know the premises held by the scientists making these claims, their sources 
of evidence, and a relevant corpus of valid facts, they cannot evaluate the cor- 
rectness of the declarations. Twelve-year-olds who can detect errors in solu- 
tions to long-division problems because they have been taught the rules of 
arithmetic are not able to transfer that skill to decide how much food and 
water the gerbils need. Put plainly, reasoning, like emotion and morality, is 
contextualized. 

Hence, schools must first teach children the facts and principles of specific 
domains — biology, chemistry, physics, psychology, sociology, mathematics, 
history, linguistics—so that they might become better analysts of the evidence 
in each of these disciplines. The specificity of the principles within each domain 
makes it difficult to generalize from one to the other. A knowledge of physics, 
for example, would lead most adults to assume that altering the sequence of 
the elementary components of a derived product should change the product. 
Geneticists, however, have learned that a few amino acids, which are the 
building blocks of proteins, can be the product of different sequences of the 
three nucleotides that comprise their foundation, as if the sequence “ran” 
retained its meaning when it was “anr,’ “rna,” or “nar.” 

A knowledge of history and sociology would lead most youth to conclude 
that because human welfare depends on the integrity and health of the society 
in which persons live, most private or public decisions should be guided by 
that principle. However, a large number of evolutionary biologists are con- 
vinced that the primary urge behind most behaviors and decisions, whether 
animals or human beings, is to maximize the inclusive fitness of self and self’s 
genetic relatives and to award less priority to the health and integrity of 
strangers or the larger community. Evaluating the truth of any empirical 
claim requires knowledge of the facts in that domain. 

The decision to celebrate critical thinking separate from a particular corpus 
of evidence required the coming together of a number of historical changes in 
our society. The most significant was the demand to honor an egalitarian 
ethic. No person is entitled to a smug feeling of superiority or to special priv- 
ilege because of his or her occupation, education, or family pedigree. This 
rebellion against any elite status began several centuries earlier but was accel- 
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erated after the civil rights movement of the 1960s. I applaud the rationale 
and consequences of this ethic, which has enhanced the dignity and privileges 
of minorities and women. But there are no free lunches; a price must be paid 
for striving to meet this ideal. 

One of the costs is a reluctance to acknowledge the presence of a value 
judgment in all educational reforms. The emphasis on critical thinking rests 
on the assumption that facts and skills have an automatic priority over feel- 
ings and sentiments. This bias characterizes the economists and political sci- 
entists who advocate Rational Choice Theory. John Rawls was celebrated by 
philosophers and political scientists because he returned the ethical notions 
of fairness and justice to theories in these disciplines. The problem with this 
premise should be obvious. Loyalty to a rationality based on facts and deduc- 
tions from those facts chases to the periphery serious consideration of the 
morality of a decision or action. Societies that either ignore the moral impli- 
cations of their actions or cannot find a consensual ethic that the majority 
applauds generate a feeling of confusion, uncertainty, or spiritual emptiness 
in their members. Thomas Jefferson noted that if a moral dilemma were stat- 
ed to a farmer and a professor, the former would more often arrive at the 
right answer. I suspect that the average plumber or truck driver would be as 
capable of judging the morality of a Congressional or Supreme Court deci- 
sion as the average lawyer, physician, professor, scientist, or chief executive 
officer. 

Most decisions by a legislature, board of directors, or school committee 
are based on the selection of a primary beneficiary from four possible candi- 
dates: the individual, their family, the society in which they live, or the world 
community. Rarely does any decision benefit all four. Furthermore, on some 
occasions, the criterion for the decision is an intuitive judgment of its moral 
worth, for example preserving the Arctic Refuge in Alaska. “The appeal to 
cool, rational analysis” Thomas Nagel wrote “is too often an excuse for refus- 
ing to listen to the clear warnings of conscience or common sense” (Nagel 
1972, 770). When Congress allocates funds to NASA to explore Mars, we 
understand that although few Americans will benefit from that expenditure 
the advanced understanding of our universe is a moral good. Most Supreme 
Court decisions cannot be defended by listing their concordance with objec- 
tive evidence on human biology or psychology but only by understanding 
the ethical beliefs of a majority in the society. 

Many contemporary educators resist the suggestion that an important 
reason for twelve to twenty years of formal schooling is that this procedure 1s 
a just, fair way to select the 1s—20 percent of the youth who will be given the 
responsibility of flying 200 travelers to a destination, operating on bodies and 
brains, defending the rights of victims in court, designing new buildings, 
investing pension funds, inventing new medicines, or filling cavities. The fact 
that only a small proportion of the members of a society are needed to fill 
these technical positions implies the politically incorrect idea of an elite. 
Because this concept 1s tainted, this truth, which older societies celebrated, 
has become ragged. 
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Each individual requires, at a minimum, a body and mind healthy enough 
to reproduce and care for the next generation, reasonable predictability of the 
future, protection from danger, social harmony, a belief that some acts and 
personal properties are always more virtuous than others, and assurance that 
others in the community share the same ethos. Preparation of the next genera- 
tion of needed technical personnel and fulfilling the six more-general require- 
ments mandates that we train each generation in the intellectual skills their his- 
torical era demands and prepare as many as possible to act with responsibility, 
honesty, and empathy in case the vicissitudes of life happen to select them 
from the very large number who could fill one of the needed roles. Most 
adults born with an intact brain and who enjoyed an adequate education are 
potentially ready to be chosen. The sad fact that not all can be plucked from 
the larger group is inconsistent with our egalitarian ethos, as well as the ideal- 
istic wish for a classless society. Hence, educators are tempted to posit ethical- 
ly neutral abstractions, like critical thinking, as primary goals because these 
beautiful ideas temporarily mute our unhappiness over the inevitability of 
social differentiation. The concept of critical thinking has the additional 
advantage of supporting our love affair with rationality as the primary basis 
for personal and public decisions. We are more likely to solve our current 
problem, however, if teachers accept the responsibility of guaranteeing that 
all adolescents, regardless of class or ethnicity, can read and comprehend the 
science section of newspapers, solve basic mathematical problems, detect the 
logical coherence in non-technical verbal arguments or narratives, and insist 
that all acts of maliciousness, deception, and unregulated self-aggrandize- 
ment are morally unacceptable. 
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CHAPTER 4 


Teaching Evidence-Based 
Citizenship 


EAMONN CALLAN 


For the last few years I have been coteaching an interdisciplinary humanities 
course for freshmen. For about three weeks in the middle of the course I take 
the podium to ponder the significance of two U.S. Supreme Court cases, 
Brown v. Board of Education and San Antonio v. Rodriguez. My lectures during 
those weeks exhibit disciplinary impurity at its most shameful or adventur- 
ous. (Which adjective you favor depends on how much or how little you 
value disciplinary purity). Legal case analysis, American social and political 
history, social psychology, constitutional theory, and political philosophy all 
jostle together as my lectures proceed. I do not tell my students which of my 
utterances are to be located in social history, as opposed to philosophy, and 
so on. They have enough trouble following me without the distraction of a 
running commentary on my many disciplinary transgressions. With the pos- 
sible exception of philosophy, I can claim to be no more than a diligent ama- 
teur in the many areas on which my teaching touches. 

But why teach amy course with such dilettantism? A more sober alterna- 
tive in a cotaught course that includes several disciplines would be to assign 
different instructors tasks that were carefully aligned with their expertise. As a 
philosopher who writes about politics and education, I might have suitably 
circumspect things to say about rival theories of justice, what these entail 
with regard to racial discrimination in state educational provision, and so on, 
all the while scrupulously avoiding saying things that might intrude on the 
academic turf of colleagues who are knowledgeable precisely where I am 
ignorant. And when my colleagues take their turns at the podium, they 
would evince a reciprocal respect for my territorial boundaries. 

The first thing we might notice about my imagined sober alternative is 
that the pedagogy in that scenario really ceases to be interdisciplinary. A pag- 
eant of experts speaking on some common themes that occupy them in their 
respective disciplines are not engaged in znterdisciplinary teaching. At best, 
they provide their students with an assemblage of thematically related disci- 
plinary fragments. One might hope that some students will have the imagina- 
tion and tenacity to create for themselves an integrated understanding from 
the fragments on display. But to suppose that what they create is to be credit- 
ed to their teachers is rather like saying that the farmer taught the chef to 
cook just by supplying the raw ingredients. 

Now this hardly gets me off the hook. I have only made a distinction 
between teaching that is genuinely (albeit perhaps recklessly) interdisciplinary 
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and teaching as the sequential presentation of disciplinary fragments that 
might (or might not) help some students to construct interdisciplinary 
understanding. I have given no reason to try to do the former. What might 
be the most obvious reason to do so also turns out to be not very helpful. 

A platitude about the growth of knowledge is that the most fertile areas of 
investigation are often found at the boundaries of adjoining disciplines. Yet 
the platitude seems more applicable to the conduct of research training than 
the teaching of introductory college courses. If the most promising areas for 
academic innovation are in fact to be found at the edges of established disci- 
plines, then good research training will draw novices in that direction. But it 
does not follow that introductory college courses must start nudging anyone 
along that route long before serious research training has started, which will 
never happen for the great majority of students in such courses. 

I want to canvas an alternative rationale for a kind of interdisciplinary 
teaching that has a proper place in the first year of college, and in high school 
for that matter, though I will also be candid about my lingering misgivings 
about such teaching. The rationale can be brought into focus through a ques- 
tion I urge students to think about in the unit on Brown and Rodriguez: To 
what extent has America succeeded in realizing the moral promise of freedom 
and equality for all its citizens, regardless of their race? For rhetorical conven- 
ience, I shall call this the Brown question. I tell my students that they will not 
learn nearly enough in our course to be entitled to an intellectually assured 
answer to the Brown question. But I also stress that this is the question that 
really matters. In the larger scheme of things, it matters far more than their 
grade or even the passions for law, history, or political philosophy that might 
be sparked in their efforts to come to grips with the question. 

The question I stress acquires its looming significance within the per- 
spective of American citizenship, and when I urge that question upon my 
students I am mindful of the fact that the great majority of them already are 
American citizens or will be in the fullness of tume. To identify with that per- 
spective is not merely to congratulate oneself for enjoying a set of legal rights 
and privileges that most human beings are denied; it is to see oneself as the 
inheritor of a remarkable experiment in republican self-government that takes 
securing the freedom and equality of all citizens as the moral imperative of 
government. And to see oneself in that light is also to accept some responsi- 
bility for making the experiment work, which means interpreting wisely and 
acting resolutely in support of the moral imperative that defines the experi- 
ment. That is not an ideologically partisan point within American politics; it 
is common ground, for example, between those who would argue that Brown 
and the Civil Rights movement that followed fully realized the color-blind 
legal dispensation that free and equal citizenship demands and others who 
would claim that progress in defeating American racism has been either mod- 
est or illusory and that new and more aggressive forms of political interven- 
tion are needed to erase the lingering effects of racial caste in America. If the 
first of these views were true (or closer to the truth than the stipulated alter- 
native), then a necessary task for the good citizen would be to conserve what 
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was accomplished in the Civil Rights movement against erosion undertaken 
in the seductive guise of group rights, the pursuit of de facto desegregation, 
or the like; if the second were (roughly) true, then to regard a color-blind 
legal dispensation as the consummation of racial equality would attest to 
grievous moral complacency at best and complicity in oppression at worst. 
Asking students to confront the Brown question is not like asking them to 
think about the causes or consequences of the Peloponnesian War; it is asking 
them whose side they are on in a struggle that their lives already encompass 
and whose importance cannot decently be evaded. In deciding on party affili- 
ation, whom to vote for, and which side to take in a host of political debates 
with compatriots, choosing well depends in no small part on good-faith 
effort to grasp the right answer, whatever answer that might be. 

If students ask the Brown question seriously, they must also construct at 
least a provisional answer on the basis of available evidence. This makes the 
specifically cognitive tasks they face in grappling with the academic material I 
present rather different than these tasks would be if the students had to think 
about them merely as novice philosophers, historians, legal scholars, or the 
like. In particular, answering the Brown question qua citizen requires an inte- 
gration of fact and value that pushes beyond the boundaries of any merely 
disciplinary understanding. 

Consider Thurgood Marshall’s claim that “unless our children begin to 
learn together, there is little hope that our people will learn to live together?”’ 
What Marshall calls “living together” can only be charitably understood as 
something like “living together as equals.” After all, whites and blacks in 
America lived together under Jim Crow without learning together, but that is 
hardly the kind of civic coexistence that Marshall envisaged. The broad politi- 
cal and legal context of Marshall’s remark suggests that he likely had at least a 
couple of things in mind under the rubric of living together as equals: Amer- 
icans would enjoy equal economic opportunity irrespective of race; they 
would also reciprocate respect for each other throughout all social institu- 
tions, whatever their race might be. Marshall’s thesis, then, is that without 
the kind of racial commingling that de facto desegregation would secure the 
promise of free and equal citizenship will remain out of reach for many 
Americans. 

Marshall’s thesis is crucial to how one answers the Brown question, and a 
huge body of social scientific research bears on the thesis. But the thesis is such 
that evidence adduced for or against it cannot be exclusively empirical. That is 
so for the obvious reason that the concepts of equal opportunity and reciprocal 
respect are open to rival interpretations, and the interpretation any one of us 
might favor is properly a matter of moral judgment. Such judgments cannot be 
warranted by any appeal to value-neutral data. For example, if cultural differ- 
ences rather than the quality of state educational provision explained much of 


1. Milliken v. Bradley, 418 U.S. 717, 783 (1974). For a circumspect defense of the educational 
and civic benefits of integration, see Clotfelter (2004). A somewhat more skeptical view is 
evident in Thernstrom and Thernstrom (2004). 
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the difference in average educational achievement between whites and others, 
would that signify inequality of educational opportunity between the relevant 
racial groups, or should we interpret cultural differences as irrelevant to the 
measurement of opportunity? If some citizens regard practices intrinsic to the 
culture of others with disdain, does this mean they fail to respect others as equal 
citizens whenever the impugned practices correlate with some socially margin- 
alized racial identity? Answers to these particular questions are presupposed by 
Marshall’s thesis because they are required by any tolerably precise understand- 
ing of what it would mean for Americans to live together as equals. But both 
answers and questions are freighted with an irreducible moral meaning, no 
matter how sophisticated and compelling our social science could become in 
addressing matters relevant to the Brown question. 

You might insist that our predicament here can still be escaped through a 
tidy division of labor between disciplines. Suppose we say that political 
philosophers rightly concern themselves with the rival interpretations to 
which concepts such as equal opportunity and mutual respect among citizens 
are open. They argue among themselves about which interpretation is best. 
Some are partisans in that debate, while others remain agnostic, either claim- 
ing that no one’s argument for any one interpretation of the concept has yet 
been decisive or that permanent, reasonable disagreement among competing 
interpretations is just what we should expect. Social scientists, for their part, 
need not dabble in intramural philosophical debate about whose interpreta- 
tion is morally best. They might take any one interpretation of equal oppor- 
tunity—call it interpretation_A—and then investigate empirically how well or 
badly Marshalls thesis withstands scrutiny, given that interpretation. They do 
not on this account have to assume that.A is a morally better interpretation 
than any of the alternatives. They can merely confine their attention to show- 
ing that ifA is the preferred option, then de facto desegregation is (or is not) 
necessary to achieving equal opportunity across America’s color lines. 

The division of labor across disciplines I have just sketched may placate 
scruples about moral neutrality in the social sciences, but it cannot by itself 
furnish the answer to the Brown question that citizens need. Citizens, unlike 
scholars, cannot just sit on the fence about what equal opportunity means on 
its most compelling interpretation. They must choose now, mustering what- 
ever knowledge and imagination they can find in order to choose well. Their 
distinctive contribution to the experiment of republican self-rule must be 
enacted in political decisions within the tumult of contemporary politics; it 
cannot be deferred to some remote future when philosophers have settled, if 
they ever do, on the best understanding of justice for free and equal citizens 
and social scientists have thoroughly explored the empirical ramifications of 
that understanding. That is why a civic education for such people will push 
against academic boundaries and the limits of our established knowledge, try- 
ing to integrate the often fragmentary and provisional outcomes of discipli- 
nary inquiry and struggling to clarify political decisions that must be made 
with much more courage and hope and less cognitive confidence than fastidi- 
ous scholars could content themselves with. 
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But now we come back to the problem of dilettantism. I have argued that 
a certain kind of interdisciplinary learning is integral to education for demo- 
cratic citizenship, using the Brown question to throw into relief the distinctive 
character of such learning. The lectures I give my freshman class are an 
attempt to foster such learning. Still, my efforts require me to engage with 
much material beyond my expertise. What worries me when I reflect on what 
I do was never better expressed than by Nietzsche in The Gay Science. After a 
characteristically dazzling passage that describes the human cost of intellectu- 
al specialization, he turns his attention to the alternative for scholars: 


But you would have it otherwise— cheaper and fairer and above all more 
comfortable—isn’t that right, my dear contemporaries? Well then, but in 
that case, you also immediately get something else: instead of the crafts- 
man and master, the “man of letters? the dexterous, “polydexterous” man 
of letters who, to be sure, lacks the hunched back—not counting the pos- 
ture he assumes before you, being the salesman of the spirit and the “car- 
rier” of culture—the man of letters who really is nothing but “represents” 
almost everything, playing and substituting for the expert, and taking it 
upon himself in all modesty to get himself paid, honored, and celebrated 
in place of the expert. (Nietzsche 1974, 323) 


If you want to put some flesh on Nietzsche’s abstractions at this point, 
imagine the contemporary “public intellectual” whom you detest the most: 
glib, intellectually reckless, sophistical, and so on. Of course, we are likely to 
admire some public intellectuals while detesting others, though I fear that in 
many cases the difference has more to do with who flatters or offends our 
prejudices rather than who really escapes or succumbs to the fate of 
Nietzsche’s “salesman of the spirit.” My worry then is that the kind of teach- 
ing I have defended is perilously susceptible to this particular corruption. 

And if I should fret about my own integrity as a teacher, I should worry 
all the more about the learning I promote when I foist the Brown question on 
my students. In the course I teach, they will become familiar with only a tiny 
fraction of the social scientific evidence that bears on the question and will 
achieve the most rudimentary grasp of the relevant complex arguments in 
political philosophy and constitutional theory. How could educational out- 
comes as scant as these entitle anyone to even a provisional answer to the 
Brown question? And if not, what could possibly be the educational point of 
my insisting on their need to confront the question? 

A thorough answer to these questions would require a lengthy (and per- 
haps tedious) disquisition on the relation between cognitive and moral virtue 
in democratic citizenship. But a passable short answer might begin from 
some prosaic facts about contemporary American citizenship without ascend- 
ing to the heights of philosophical arguments about citizenship. A recent 
study by political scientists John Hibbing and Elizabeth Thiess-Morse yields 
the following rather depressing description of how most Americans perceive 
their own politics: 
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[T]he kind of government people want 1s one in which ordinary people 
do not have to get involved. We show that people want to distance them- 
selves from government not because of a system defect but because many 
people are simply averse to political conflict and many others believe 
political conflict is unnecessary and an indication that something is wrong 
with governmental procedures. People believe that Americans all have the 
same basic goals, and they are consequently turned off by political debate 
and deal making that presuppose a lack of consensus. People believe that 
these activities would be unnecessary if decision makers were in tune with 
the (consensual) public interest rather than the cacophonous special inter- 
ests. Add to this the perceived lack of importance of most policies and 
people tend to view political procedures as a complete waste of time. 
(Hibbing and Theiss-Morse 2002, 7) 


To invent an account of political attitudes less in keeping with conceptions 
of democratic citizenship, such as mine, that demand strenuous cognitive and 
moral effort from citizens would be difficult. Moreover the authors’ evidence 
suggests that the attitudes they describe are little affected by education, at 
least as it is currently practiced (Hibbing and Theiss-Morse 2002, 146-147). 

The very attempt to encourage an evidence-based approach to the Brown 
question might seem merely quixotic given both the empirical reality of 
American citizenship that Hibbing and Thiess-Morse describe and the negli- 
gible progress my students can be expected to make in answering the ques- 
tion as a consequence of my teaching. But an important distinction to bear in 
mind is between the cultivation of civic virtue and the mitigation of civic vice. 
I do not expect that students who may have just come to appreciate that con- 
troversy about equality and race in America is entangled with intellectually 
deep questions about facts and values are now on the threshold of a consis- 
tently reflective and informed citizenship. That being so, only delusions of 
grandeur could induce me to think of myself as cultivating Socratic civic 
virtue. But I can still see educational value in the more elementary endeavor 
of countervailing some of the very civic vices that Hibbing and Theiss-Morse 
expose—for example, the smug moral complacency of those who think that 
the common good is transparently clear to all of us, so that only disruptive 
“special interests” would challenge the status quo. I am not at all sure if I 
enjoy much success even relative to that more modest goal. But the hope that 
I can achieve something in that regard is enough to keep me teaching in dan- 
gerous proximity to salesmen of the spirit. 
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CHAPTER 5 


Learning to Reason 
about Evidence and 
Explanations: Promising 
Directions in Education 


TINA GROTZER 


INTRODUCTION 


Recently people were sickened by eating tainted spinach. Government authorities 
rushed to gather evidence about the source and scale of the problem and whether it 
extended to other leafy green vegetables. How would you decide whether to feed leafy 
green vegetables to your child? 


Sometimes when you put on sunscreen, you break out in an angry red rash. You 
haven’t noticed any obvious pattern in which sunscreens cause a vash. How can you 
figure out what to do? 


What does it mean to encourage a scientifically literate population? We live in 
an age where an abundance of information is readily available at our finger- 
tips. How do we help children grow up to be critical consumers of scientific 
information and to reason effectively about it in the service of better lives and 
a healthier planet? What about creating the next generation of scientists? 
How do we open the door so that more students can pursue science? These 
are questions that science education researchers strive to answer and K-12 
educators grapple with every day. 

This paper considers the difficulties students have in reasoning about 
scientific evidence and the current discourse in K-12 education research and 
practice about the puzzles and what appear to be promising directions. It 
draws from the research literatures in cognitive science, development, and 
science education to provide an overview of the issues and to illuminate the 
nuanced and highly challenging problem space of scientific reasoning and 
evidence evaluation. Finally, it offers suggestions for the focus of our collec- 
tive efforts for improving practice toward developing a more scientifically lit- 
erate population. Given space considerations and the extensive research liter- 
ature on this topic, this paper samples the available research literature but is 
by no means exhaustive. 
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WHAT MAKES REASONING ABOUT SCIENTIFIC 
EVIDENCE SO DIFFICULT? 


Educating for scientific literacy requires an answer to what makes reasoning 
about scientific evidence so hard. An expansive literature addresses this ques- 
tion, ranging from research on why people attend to certain types of informa- 
tion over others in a perceptual sense (e.g., Mack and Rock 1998) to why cer- 
tain information is successful in capturing our attention (e.g., Sunstein 2002) 
and how default biases and heuristics make it difficult to reason well in any 
given situation (e.g., Nickerson et al. 1985). It includes research on people’s 
difficulties with proportional reasoning, probabilities, and statistics (for a 
review, see Piatelli-Palmarini 1994). Another body of research focuses on rea- 
soning about complexity—not only the requisite skills and patterns of think- 
ing but ways of dealing with the cognitive load of many interrelated, typically 
dynamic concepts (e.g., Dorner 1989; Feltovich et al. 1993). 

In the event that one surmounts these challenges in gaining, sustaining, 
and focusing attention and helping the public successfully reason about the 
scientific evidence at hand, the literature in the field also invites us to consid- 
er the sociology of the problem space. A diverse literature in its own right, it 
considers questions such as what is the nature of science and how do the pat- 
terns in science interact with how people attend, believe, and understand it? 
For instance, if science is a process of trading up for more powerful explana- 
tory models, as Thomas Kuhn (1962) suggests, where does this leave the pub- 
lic, who might hold an accumulation model of how science works, in trying 
to reason about the available scientific evidence? From the perspective of the 
public, science might seem to change its mind a lot. The literature in the field 
also addresses how scientists handle uncertainty (e.g., Zehr 1999), how dis- 
agreements are vetted, and how local knowledge and “expert knowledge” 
interact to inform a problem space (e.g., Jasanoff 1997; Wynne 1992). Highly 
public incidents that draw upon scientific knowledge, such as the spinach 
scare outlined above, bring the public “along for the ride,” which invites all 
sorts of opportunities to misunderstand the evidence and doubt the process 
of generating it. Highly visible cases where the scientific establishment, 
through failures of the disciplinary means that vet good science from bad, 
was initially wrong lead to further doubt and mistrust. An example is the 
well-publicized case of Judah Folkman, who proposed a radically different 
theory of cancer growth that was initially spurned by his colleagues. Folkman 
persisted in developing the theory of angiogenesis and to change the course 
of cancer treatment today. However, it’s easy for the public to take away the 
simpler message that the scientific establishment was wrong than to evaluate 
the broader set of cases among which are many instances where the processes 
of science policed itself to keep bad science out of the public domain. 

Given these challenges, it is no wonder that achieving a scientifically liter- 
ate population 1s a focus of concern. We know a fair amount about the prob- 
lem space and why it is so complicated. However, this knowledge doesn’t 
lead to easy answers about what we should do. The sections that follow 
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explore some of the key areas of research about scientific reasoning that illu- 
minate the direction of research and practice in education. While these direc- 
tions offer promise, the concluding section reflects upon these directions and 
considers what else we might be doing in light of how complicated the prob- 
lem space is. 


What Is Needed to Get Us to Change Our Minds? 
Confirmation Bias and Other Tendencies 


How do people respond to different types of evidence? In a seminal work, 
Deanna Kuhn and colleagues (1988) reviewed a wide-ranging set of studies 
and conducted targeted studies related to how people reason about evidence. 
They found that people typically don’t change their minds and that they have 
a number of difficulties reasoning about evidence. For instance, Kuhn et al. 
found that people did not distinguish between the evidence and the theory 
itself. When evidence was discrepant with a theory, people failed to acknowl- 
edge it, or they adjusted or ignored it. In cases where they only moderately 
believed the theory, they often adjusted the theory to fit the evidence without 
considering the implications of doing so. They also had difficulty perceiving 
instances of a nonoperative variable or an operative variable that led to an 
outcome opposite of what they expected. These studies are part ofa line of 
research leading to what has been called “confirmation bias”’—the tendency to 
use evidence to confirm one’s existing beliefs. Confirmation bias has been 
widely studied and corroborated in the field. For instance, Kahr and col- 
leagues (e.g., Schunn and Kahr 1993; Mahr et al. 1993) found subjects are less 
skeptical of self-generated hypotheses than those generated by others and 
more rigorously tested other-generated hypotheses. 

However, further discourse in the field suggested that when viewed in 
context and with a nuanced understanding of the nature of scientific reason- 
ing, subjects’ responses in earlier studies might make scientific sense (e.g., 
Karmiloff-Smith 1984; Koslowski 1996). Koslowski (1996) argued that scien- 
tists do not work in a theory vacuum, so studies asking subjects to reason 
about theoretically impoverished situations are incomplete and might distort 
what would be considered scientifically appropriate in a given situation. Like 
scientists who use “working hypotheses” that don’t fit all available data to 
reduce processing demands and to organize data in order to search for pat- 
terns, people might hold on to an initial theory because of a temporary lack 
of a better one. Koslowski raised the question of when theory modification is 
scientifically legitimate. The real issue, she argued, is not whether one seeks 
confirmation or disconfirmation but whether one considers plausible alterna- 
tive hypotheses. Science often proceeds by reformulating rather than discard- 
ing a theory because of disconfirming evidence. Dunbar (1995) studied scien- 
tists in the lab and found that experienced scientists tend to pay great 
attention to inconsistent results but that most often they changed particular 
features of their hypotheses rather than discarding it entirely. On the other 
hand, he found less confirmation bias in experienced scientists than in inex- 
perienced ones but that the former often discarded useful data. 
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Chinn and Brewer (1998) studied how students responded when con- 
fronted with anomalous data or evidence that contradicts their current theo- 
ries of the world. They found that students responded in seven ways. 
Students might 1) ignore the data; 2) reject the data (often offering a different 
explanation); 3) exclude the data from the domain of the specific theory; 

4) hold the data in abeyance; s) reinterpret the data while retaining the theo- 
ry; 6) reinterpret the data and make peripheral changes to the theory; and 

7) accept the data and change theory A, possibly in favor of theory B. 
However, for each response type, Chinn and Brewer offered examples from 
historical cases of scientific reasoning to show that scientists also use these 
response patterns. Karmiloff-Smith (1984) found that children temporarily 
ignore disconfirming data until they form a solid theory; then they turn to 
those data again to try to come up with a new theory to explain them; then 
they generate a unifying theory. 

Reaching a conclusion opposite of what one originally expects may also 
depend upon the tools that one has within his or her repertoire. A greater 
focus on concepts that support the interpretation of research data, such as 
statistical inference, random sampling, reliability, and regression to the mean, 
can support effective modeling and reasoning about data. These concepts 
typically bridge the science and math curricula and might not receive atten- 
tion in either place. Petrosino and colleagues (2003) argue that concepts of 
variability, which are central to data modeling, are given short shrift in school 
instruction. They report on an eight-week unit in which fourth graders were 
given the tools to think about what was suggested by the distribution of their 
data and thus successfully coordinated the data with conjecture to reach a 
finding opposite of what they initially expected. 

This key area of research points to the importance of nuance in how evi- 
dence is interpreted. While important general patterns of reasoning are 
involved, the effective interpretation of evidence relies on many contextual 
factors. This suggests that mastering and being able to transfer such skills 
requires diverse, contextualized opportunities from which learners can dis- 
cern more expert patterns of engagement. 


When Does One Thing Actually Cause Another? Causality as Covariation 


Consider how you might think about the following questions: “What is caus- 
ing a rash sometimes when you put on sunscreen but not other times?” Or, 
“What might be causing you to feel ill sometimes after eating?” One of the 
first things that most people do is to search for a pattern that goes along with 
or co-occurs with the outcome. For instance, you might realize that you only 
get a rash when you wear a sunscreen strength of 40 SPF or higher. Or per- 
haps you get ill after eating certain kinds of foods or at certain times of the day. 
How we detect that a cause and effect relationship exists has a profound 
impact on what we view as relevant evidence. Kuhn and colleagues (1988) 
found that people relied on covariation of two or more variables to suggest a 
causal relationship. The variables might covary in different ways (“When I 
drink less water, I feel more tired” or “The more caffeine I drink, the more 
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productive I am”) or in more than one direction (“The more caffeine I drink, 
the more productive I am until I crash and get nothing done”). They also 
found that people tended toward false inclusion—to view any covariate as 
causal even in cases where it was merely correlational. Finding covariation 
with one variable, subjects were unlikely to test other uncontrolled variables. 
For instance, you might decide that sunscreen of SPF 40 and higher is the 
culprit in the rash, but you put on sunscreen of SPF 40 and higher only when 
you spend long periods of time in intense sun. Perhaps the high SPF level 
isirt what is causing your rash but long periods of exposure to intense sun. 

Overapplied covariation can lead to assumptions and erroneous conclu- 
sions. For example, the College Board established a high positive correlation 
between students who took algebra in eighth or ninth grade and those who 
went to college. The U.S. secretary of education interpreted this to mean that 
courses in mathematics, including algebra, were the gateway to college 
(Bracey 1998). Yet, we dort know if a third variable, perhaps related to par- 
enting or ability level, led to both outcomes. Further, while covariation in 
terms of temporal and spatial contiguity can be an important cue to the possi- 
bility of a causal relationship, as causality becomes more complex, relying on 
co-varying, contiguous variables can result in shortsightedness because some 
causes are spatially and temporally remote from their effects (Shultz and 
Mendelson 1975). Both of these kinds of errors can be detected in people’s 
reasoning—the tendency to assign a causal relationship to one that is merely 
correlational and the tendency to look for local causes and effects and to miss 
spatially and temporally remote ones. 

Research shows that people do use covariation, but this does not appear 
to be the whole story. Infants use contingency, expecting that objects act on 
one another only if they touch and that action is not enacted from a distance 
(Borton 1979; Leslie 1982, 1984; Leslie and Keeble 1987; Oakes 1993; Spelke 
et al. 1995; Van de Walle and Spelke 1993), and clear, age-related increases 
occur in people’s ability to use contingency data to cue the possibility of a 
causal relationship. By preschool, children have gained greater understanding 
of the contextual nuances of applying temporal and spatial cues. They are less 
likely to pick a spatially remote event as a cause (Koslowski 1976; Koslowski 
and Snipper 1977; Lesser 1977) and when spatial contiguity cues conflict with 
temporal cues, children override the former in favor of the latter (Bullock and 
Gelman 1979). Time delays between causes and effects introduce difficulties 
for preschoolers (Siegler and Liebert 1974), but by five years of age children 
are able to deal with time delays when the delays have an identifiable explana- 
tion (Mendelson and Shultz 1976); for instance, the physical effects that 
result from illness after contact with a contamination (Kalish 1997). Adults 
are able to override temporal and spatial contiguity cues when the specific 
context—for instance, the possibility of a complex interaction between two 
competing causes —suggests the need to (Michotte 1946/1963). Adults also 
are more likely to have and therefore use information about the contingency 
relations between two events —how often they are likely to co-occur (e.g., 
Kelley 1973; Nisbett and Ross 1980). 
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Research also underscores how nuanced people’s reliance on covariation 
data are. Siegler and Liebert (1974) found that when they varied the degree of 
covariation (100 percent to 50 percent) between events and introduced vari- 
ability into the temporal contiguity (immediate versus five-second delay), 
eight- and nine-year-olds were more sensitive to the lack of perfect covaria- 
tion than five-year-olds, perhaps due to the distraction of time delays and 
being less likely to notice the lack of perfect covariation. Gopnik et al. (2004) 
have argued that young children can override imperfect correlation, may be 
accepting of the probabilistic context, and may indeed reason as Bayesians, 
summing across experiences to discern frequencies of particular patterns. 
Investigations of how people use contingency data in determining cause and 
effect relationships suggests an age-related increase in the accuracy of third- 
and seventh-grade children’s and adults’ judgments of a cause-effect relation- 
ship (Shaklee and Goldston 1989; Shaklee and Mims 1981). 

Is covariation all that people use in determining a causal relationship? 
Consider the following two statements: 1) A high correlation exists between 
the numbers of churches and the crime rate in the United States; 2) A high 
correlation exists between growing up in certain neighborhoods and the like- 
lihood that one will engage in crime. If you are like most people, you consid- 
ered what causal mechanism might be in play in each case and were less likely 
to assign a causal link between churches and crime rate as between growing 
up in a tough neighborhood and engaging in crime. Research suggests that 
people do not engage in Popperian mechanism-independent evaluation of 
evidence. In the sunscreen example, you might also notice that each time you 
got the rash the sunscreen came out of a white bottle. However, most of us 
would rule out bottle color as a plausible causal mechanism no matter how 
highly correlated it is. 

Even preschool children attend to mechanism in their causal explanations 
(e.g., Bullock 1979; Baillargeon et al. 1981). Despite a clear recognition of 
covariation as a cue to a potential causal relationship, even young children are 
sensitive to mechanism (Bullock 1979) and as early as four and five years of 
age can override spatial discontiguity if plausible mechanisms exist. Even 
three-year-olds aren’t indifferent to causal mechanisms; they just do not have 
much information about them (Baillargeon et al. 1981). Preschoolers realize 
that seemingly uncaused events required explanation, even if they are unable 
to specify the details (Bullock 1984, 1985; Corrigan 1995; Gopnik et al 2004; 
Shultz and Kestenbaum 1985). 

This area of key research suggests that our biases and our abilities might 
be separate, that while we are inclined to confuse correlation and causation, 
we are better able to assess causation than we always do. This suggests that 
helping people reflect upon their tendencies when evaluating evidence and 
cueing them to instances when they need to be alert to other variables should 
be effective. 
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There’s No Simple Way to Handle Complexity: 
Reductive Biases and Causal Default Patterns 


In the process of analyzing evidence, one implicitly makes decisions about 
how to bound the problem space. An unbounded space is untenable in many 
reasoning contexts; yet, as with spatially and temporally remote causes, more 
extended searching is sometimes needed. Similarly, one makes decisions 
about how to characterize the inherent features and complexity. In the same 
spirit of efficiency and comprehensibility, when dealing with complexity peo- 
ple tend to reveal “reductive biases.” Feltovich et al. (1993) identified charac- 
teristics of concepts or situations that cause difficulty for most people and 
found that people tend to simplify phenomena in a type of reductive bias. For 
instance, people often reduce dynamic phenomena to static snapshots and 
continuous processes to discrete steps. 

People reduce information into sets of heuristics as well. For instance, 
people tend to use an “availability heuristic”? when sampling information 
(Tversky and Kahneman 1982). We sample the most available information 
from memory. Drawing conclusions from frequency of experience tends to 
serve us well but can cause errors by biasing the sample (examples include 
remembering events with shock value and so overestimating their occur- 
rence; Sunstein 2002). Research on “intuitive toxicology” reveals that people 
hold assumptions about the nature of chemicals that are quite divergent with 
what experts believe (Sunstein 2002). Laypeople tend to believe that any 
human-made chemical is bad while any substance from nature is benevolent, 
whereas experts are more interested in the dosage and recognize that chemi- 
cals in nature can be as or more harmful than manufactured ones. Experts 
view naturally occurring radon as a much greater cancer threat than human- 
made sources of radiation. How such biases can influence public understand- 
ing and direct public sentiment in reasoning about scientific evidence is 
exemplified in the current look back at Rachel Carson’s effectiveness in the 
banning of DDT, the lack of scientific evidence to support the banning, and 
the subsequent malaria deaths (Sunstein 2002; Tierney 2007). 

How we reach conclusions about the world around us is also impacted by 
how we structure causality. Grotzer and colleagues (Grotzer 2004; Perkins 
and Grotzer 2005) identified nine simplifying causal assumptions that people 
tend to make in their explanations. People assumed 1) linearity as opposed to 
nonlinearity; 2) direct connections between causes and effects without inter- 
vening steps or indirect connections; 3) unidirectionality as opposed to bidi- 
rectionality; 4.) sequentiality as opposed to simultaneity; 5) that causes and 
effects are obvious and perceptible as opposed to nonobvious and impercep- 
tible; 6) active or intentional agents as opposed to nonintentional ones; 

7) determinism—where effects must consistently follow “causes” or the 
“cause” is not considered to be the cause as opposed to probabilistic causa- 
tion; 8) spatial and temporal contiguity between causes and effects as 
opposed to spatial gaps or temporal delays; 9) centralized causes with few 
agents — missing more complex interactions or emergent effects as opposed 
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to decentralized causes or distributed agency. Substantial support for these 
tendencies exists in the research literature (e.g., Feltovich et al. 1993; Ferrari 
and Chi 1998; Wilensky and Resnick 1999). For instance, Resnick (1996) 
refers to the “centralized mindset” and Driver et al. (1985) identified a tenden- 
cy toward simple linear causal explanations. 

Reductive biases and heuristics can help us quickly and efficiently process 
information. Therefore, in some respects they make sense. In other contexts, 
however, we might miss parts of the causal story and make uninformed deci- 
sions. This key area of research invites reflection on the tendency in education 
to reduce complex phenomena to simpler versions and raises questions about 
how we can help learners develop awareness and a reflective stance on the 
heuristics and causal reasoning patterns that they employ. 


Summary Points 


The sum of the research in these key areas suggests the difficulty of under- 
standing the nature of scientific reasoning and evidence evaluation without 
the nuanced context that surrounds such reasoning and evaluation. Strategies 
that might seem simplistic, reductive, or misguided in one context might fit 
with scientific practice in another context. Brem and Rips (2000) found that 
people’s ability to reason about evidence was significantly better in informa- 
tion-rich than information-poor conditions. Participants in their study tend- 
ed to prefer evidence, but the value they placed on explanations rose signifi- 
cantly when they believed information was scarce. Participants in rich 
information contexts referred to evidence twice as often as those in poor 
information contexts. At the same time, people engage in simplifying strate- 
gies that complicate their ability to evaluate evidence well. People appear to 
hold biases that compete with expert evidence-evaluation skills. While gener- 
alizable skills do apply across evidence-evaluation contexts, how those skills 
are enacted in a given situation depends largely on the nuances of the con- 
text. Researchers have written about this as the coordination of two problem 
spaces (e.g., Kahr and Dunbar 1988; Kuhn et al. 1988). What does all this 
suggest for research on understanding causality and reasoning about evi- 
dence? Further, what does it suggest for educational practice? 

The developmental research community has responded to the difficulties 
of the problem space by carefully controlling task demands so that young 
children can reveal their emerging competencies. Broadly, this work reveals 
young children to be far more competent in their reasoning than earlier 
research suggested. These studies are useful as a suggestion of what aspects of 
more complex competencies children hold at given ages. They also suggest 
what might be possible in real-world contexts. However, these competencies 
don’t typically carry over to scientific reasoning tasks that feature authentic, 
ill-structured, and compound forms of complexity within a given problem 
context (for a review, see Grotzer 2003). The response of the educational 
research community has been to study more-authentic context-rich tasks. 
Research investigations that reflectively build our knowledge of evidence 
evaluation skills through a combination of context-lean and context-rich tasks 
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and specifically analyze them through that lens are likely to offer the deepest 


insights into how people reason about scientific evidence. 


EFFORTS IN K-12 EDUCATION AND RESEARCH TO IMPROVE 
SCIENTIFIC EVIDENTIAL REASONING 


Evidence evaluation holds a clear place in the K-12 science standards. The 
standards call for the ability to reason about evidence in relation to scientific 
explanation. For instance, they state that students should “use evidence to 
generate explanations, propose alternative explanations, and critique explana- 
tions and procedures” (National Research Council 1995). Further, students 
should “develop descriptions, explanations, predictions, and models using 
evidence” (National Research Council 199s). The standards place a particular 
focus on the relationship between evidence and explanations by requiring 
that students be able to “think critically and logically to make the relation- 
ships between evidence and explanations” and be able to account for anom- 
alous data. The standards also recognize some of the puzzles involved in 
teaching children to reason well about evidence. For instance, they call for 
the ability to “recognize and analyze alternative explanations and predictions” 
(National Research Council 199s). They acknowledge some of the difficulties 
that students have in evaluating evidence and attempt to alert teachers to 
these difficulties. For instance, “teachers of science for middle-school stu- 
dents should note that students tend to center on evidence that confirms 
their current beliefs and concepts (i.e., personal explanations), and ignore or 
fail to perceive evidence that does not agree with their current concepts” 
(National Research Council 1995). Of course, what the standards call for and 
what happens in K-12 classrooms are two different pictures. The following 
paragraphs consider some of the promising directions that science education 
research has taken and some of the difficulties of enacting these innovations 
in the classroom. 


Inquiry-Based Science 


The focus in the standards is largely on the evaluation of evidence in the con- 
text of scientific inquiry—as opposed to evaluating evidence in the process of 
assessing research in the popular press and elsewhere. For instance, the stan- 
dards focus on “the ability to conduct inquiry and develop understanding 
about scientific inquiry, including asking questions, planning and conducting 
investigations, using appropriate tools and techniques to gather data, think- 
ing critically and logically about relationships between evidence and explana- 
tions, constructing and analyzing alternative explanations, and communicat- 
ing scientific arguments” (National Research Council 1995). 

Increasingly, inquiry-based science approaches, in which students pursue 
questions through experimentation, are finding their way into K-12 class- 
rooms. While inquiry-based instruction is typically hands-on, the two 
approaches are not synonymous, with “hands-on” including teacher-posed 
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activities and recipe-like lab experiences that are not necessarily “minds-on” 
too (Driver et al. 1985). The move toward inquiry-based science recognizes 
the important epistemological knowledge that students gain in such contexts, 
the increased likelihood of transfer, and the ability to deal with ill-structured 
real-world problems. Inquiry-based science also offers students the opportu- 
nity to learn scientific reasoning —how nuanced scientific reasoning can be 
and what aspects of scientific reasoning generalize across different contexts 
(Kuhn et al. 1992). 

What do we know about the impact of inquiry-based instruction? 
Students have difficulty formulating research questions and plans for investi- 
gating them (Krajcik et al. 1998). Zimmerman (2000) summarized fourteen 
self-directed experimentation studies and found the following patterns: In 
general, third to sixth graders often generated uninformative experiments; 
made judgments that were based on inconclusive or insufficient data while 
ignoring inconsistent data and disregarding surprising results; attended to 
causal factors but not noncausal factors; were unsystematic in recording data, 
outcomes, and plans; and were influenced by and had difficulty disconfirm- 
ing prior beliefs. 

However, this research also suggests some promise. Students did learn 
new strategies, but these appear to live side by side with old ones, not neces- 
sarily replacing them. Masnick and Kahr (2003) found that second- and 
fourth-grade children could both propose and recognize potential sources of 
error before they could design unconfounded experiments. They used evi- 
dence to guide their reasoning, making predictions and drawing conclusions 
based on the design of their experiments, and they were sensitive to the con- 
text of reasoning: they differentiated the role of error in relative and absolute 
measurements. Masnick and Klahr (2003) argued that long before children 
have acquired the formal procedures necessary to control error, they have a 
surprisingly rich—albeit unsystematic— understanding of its various sources. 

Further, the research on how good learners and bad learners think about 
evidence suggests specific ways that we might help all learners to learn better. 
Schauble et al. (1992) found that good learners generate and state many more 
alternative hypotheses, their experiments are more controlled, and they do a 
more extensive search of the problem space. They are more systematic— 
recording results and goal-oriented plans. When students hold more-sophis- 
ticated conceptual models, they are more likely to try to make sense of dis- 
confirming or surprising evidence (e.g., Dunbar 1993). 

Helping students engage in authentic, inquiry-based science gives them 
the opportunity to learn scientific reasoning and also increases the knowledge 
demands on teachers in numerous ways. To be effective, teachers need 
nuanced epistemological knowledge, a considerably greater content know!- 
edge basis upon which to respond to puzzles and questions that arise, and 
much more sophisticated pedagogical knowledge (to name but a few of their 
needs). What students end up learning may indeed be more valuable, but 
what is learned and the quality of learning is typically far more divergent than 
with traditional, didactic teaching approaches and content costs may accrue if 
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the teacher is not highly skilled in figuring out how to weave content explo- 
ration into the inquiry context. 


The Epistemology of Science 


Engaging in effective inquiry-based science practices is more than knowing 
how to carry out experiments, however. Sandoval and Reiser (2004) have 
argued that students need to understand the epistemological commitments 
that scientists make—the processes they value for generating and validating 
knowledge. They call for foregrounding these commitments in the context of 
inquiry-based approaches. This goes beyond helping students learn to “think 
like scientists.” The aim here is to help students learn the nature of science and 
the epistemological underpinnings of the discipline. The national science 
standards also address the importance of knowledge of the epistemology of 
science. They call for an understanding of the ways of knowing and finding 
out in the discipline; for instance, “scientists develop explanations using 
observations (evidence) and what they already know about the world (scien- 
tific knowledge). Good explanations are based on evidence from investiga- 
tions.” And “scientific explanations emphasize evidence, have logically consis- 
tent arguments, and use scientific principles, models, and theories. The 
scientific community accepts and uses such explanations until displaced by 
better scientific ones. When such displacement occurs, science advances” 
(National Research Council 1995). 

Research suggests that peoples’ ability to reason about evidence may be 
limited by their epistemological development (Lederman et al. 2002; 
Sandoval 2003, 200s) and that students with more epistemological knowl- 
edge generally perform better in science (e.g., Linn and Songer 1993). This 
type of knowledge provides the broad context for everything else learned in 
science and puts students in a better position to join scientific communities. 

But understanding the epistemology of science isn’t just for future scien- 
tists. Making sense of the wealth of scientific data around us and trusting the 
scientific enterprise as a whole requires an understanding of how that knowI- 
edge is generated. The common view of science is often quite different from 
and leads to a different set of expectations than that held by scientists 
(Chalmers 1999). For example, scientists realize that conclusions are always 
tentative, that science is not a steady process of the accumulation of facts but 
that as evidence no longer fits a prevailing model, we trade up for more 
explanatory models in a Kuhnian paradigm shift (Kuhn 1962). What counts as 
evidence also can change as part of a paradigm shift. However, if the general 
population expects scientific knowledge to accumulate, trading up seems more 
like scientists are changing their minds, and, if that is part of science, why 
would you place your trust in scientific evidence and the conclusions scientists 
draw from it? Scientists need to consider carefully how to represent and com- 
municate uncertainty (Zehr 1999), and the general population needs to under- 
stand what that uncertainty means in the context of a scientific framework. 

In practice, helping students learn the epistemology of science holds a 
number of challenges: 1) understanding the tacit assumptions that scientists 
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make; 2) preparing teachers to understand those assumptions; and 3) finding 
ways to make epistemological assumptions accessible at the right level for 
students without reducing them to a stereotyped set of steps. 

Unpacking the tacit assumptions that scientists actually make has been an 
important line of research (e.g., Dunbar 1995). Scientific discovery and rea- 
soning is a highly nuanced and opportunistic endeavor. In telling the story of 
the discovery of binary pulsars for which he eventually won a Nobel Prize, 
Russell Hulse (2003) talks of the process leading up to the discovery. He 
explains the data collection and careful note taking on the behavior of pulsars 
and the “error” in his data that emerged first as an annoyance and, after fur- 
ther reflection, as an interesting aberration, and, ultimately, as a pattern 
describing a previously unidentified phenomenon. His story underscores the 
nuanced attention to certain kinds of patterns and the reasoning processes to 
follow that characterize scientific reasoning —nuance that can be difficult to 
unpack and capture in understanding the scientific process. 

In most cases, K-12 teachers do not hold the epistemological assumptions 
of a scientist. This is particularly an issue with K—s teachers who may have lit- 
tle or no science background and are expected to teach all the disciplines, 
understand developmental issues, work with students who have different 
abilities, disabilities, and skill levels, may not speak English as their primary 
language, and so on. However, epistemology can also be an issue for middle 
and high school teachers who might be well versed in content but unfamiliar 
with how knowledge is generated in the sciences. Even if they once knew, the 
predominant modes of inquiry shift and change over time with new tools. 
For instance, in genetics, computers now make it possible to randomly run 
huge numbers of gene sequences searching for a match rather than using a 
theory-oriented approach to eliminate certain sequences first. Teachers need 
these understandings if they are to help students learn them. 

Finally, making nuanced assumptions visible and accessible without 
reducing them to a set of stereotyped steps is difficult. Often scientific 
inquiry is taught in K-12 classrooms as a set of steps to follow; for example, 
“the scientific method” is often taught this way. In the first step, students are 
told to “get a hypothesis”—leaving out the entire idea-generation phase of 
scientific discovery! Lessons might be designed so that students engaged in 
inquiry are doing the inquiry either with or without explicit reflection on the 
processes. These stereotyped versions of science impact what the population 
expects of science as a field. Yet, as Bauer (1992) has argued, what the general 
population believes about the nature of scientific thinking includes many 
popular “shoulds” that would impede scientific progress. For instance, if sci- 
entists published all their data, the community would be mired in unsound 
or misleading data as well as that which might be constructive. 

One approach is to teach about the nature of science by exploring histori- 
cal and current-day examples of scientists and scientific reasoning. While this 
has the benefit of inviting students to consider the nuances of the field, such 
an approach would need to pay careful attention to how the cases are present- 
ed. As with the Folkman case, it’s easy to skew students’ sense of the disci- 


TEACHING EVIDENCE-BASED DECISION MAKING 


pline by sharing only examples that stand out for one reason or another, such 
as those that warrant historical recognition. Revolutionary science that cuts 
across disciplinary boundaries or that shifts the current paradigm is far less 
common than the everyday science that involves solving smaller-scale puzzles 
and slogging through data. 

Despite the challenges, understanding epistemology is clearly a key to 
developing a population that knows how to think about scientific evidence. 
Students can learn to think about epistemological issues (Smith et al. 2000), 
and explicit reflection on epistemology results in more-informed views of the 
nature of science (Khishfe and Abd-El-Khalick 2002). A promising direction 
in science education research investigates students’ ability to gain from the 
use of technology-based, epistemic tools that scaffold their framing of the 
epistemology (e.g., Bell and Linn 2000; Sandoval and Reiser 2004; 
Scardamalia and Bereiter 2004). In classrooms using these resources, stu- 
dents have, for instance, demonstrated the ability to negotiate the terms of 
explanations, engage in planful investigation (Schauble et al. 1991), and evalu- 
ate whether evidence fits with their explanations or not (Sandoval and Reiser 
2004). 


Science as Argumentation 


The science education research community has shown growing interest in 
argumentation as a central scientific practice that students should learn (e.g., 
Driver et al. 2000; Kuhn 1993; Sandoval and Millwood 2005). The mere pres- 
entation of contradictory evidence is not enough to get students to change 
their minds (Chinn and Brewer 1998). Driver et al. (2000) argue that the 
practices of science teaching need to be reconceptualized so as to portray sci- 
entific knowledge as socially constructed—emphasizing the role of argumen- 
tation in science. 

The standards outline a clear role for scientific discourse and argumenta- 
tion in science classrooms. They call for teachers to “structure and facilitate 
ongoing formal and informal discussion based on a shared understanding of 
the rules of scientific discourse” (National Research Council 1995) and for 
students to develop the 


ability to engage in the presentation of evidence, reasoned argument, and 
explanation comes from practice. Teachers encourage informal discussion 
and structure science activities so that students are required to explain and 
justify their understanding, argue from data and defend their conclusions, 
and critically assess and challenge the scientific explanations of one anoth- 
er. (National Research Council 1995) 


How argumentation is carried out matters. Debate about why certain 
explanations are better than others appears to be a critical component in 
developing epistemic criteria (Rosebery et al. 1992; Sandoval and Reiser 
2004). Hogan (1999) found that students who engaged in lessons designed 
to encourage “metacognitive, regulatory, and strategic aspects of knowledge 
co-construction” were subsequently better able to articulate their collabora- 
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tive reasoning processes than the students in control classrooms. Further, 
when one student uses an evidence evaluation strategy in a discussion, the 
strategy is more likely to be used by his or her classmates (Anderson et al. 
2001; Pluta and Chinn 2007). Unfortunately, most teachers do not provide 
many opportunities for group or class discussion— expressing uncertainly 
about how to support such discussions, they opt instead to lecture (Newton 
et al. 1999). Student discussion, when it does occur, tends to focus on proce- 
dural aspects of the practical work rather than the actual science. 


Modeling 


The epistemology of science involves thinking about the explanatory power 
of a model in light of the available evidence (e.g., Giere 1988; Hestenes 1992). 
Scientific knowledge is generated by discarding models that no longer fit the 
evidence and then trading up for more powerful models. Consider how dif- 
ferent this is from typical “school science” where students are taught a “right 
answer” and are not always taught the rationale for that explanation. 

Models are a natural extension of a classroom discourse that involves 
argumentation and teaching of the epistemology of science. Models are 
debated and defended and in this way render students’ thinking visible 
(Lehrer and Schauble 2006) to the person espousing the model, to other stu- 
dents, and to the teacher. The models become an artifact of the sociocultural 
process in the classroom. Others have argued that models assist in the trans- 
fer of learning (Clement 2000) and in conceptual change (Gobert 2000). 

How students think about models bears on how they use them to reason 
about evidence. Harrison and Treagust (2000) argue that most students 
believe in a one-to-one correspondence between a model and reality; there- 
fore, students need to learn that all models fit in some ways but not in others. 
Perception of models appears to be rooted in students’ understanding of the 
epistemology of science. Chittleborough and colleagues (2005) found that 
many students do have a good understanding of the role of models in the 
process of science and appreciate the multiplicity and representational yet 
changing nature of models. While some students have a fascination for true 
facts and single and correct models, others exhibit more sophisticated episte- 
mologies of science. Chittleborough and colleagues (2005) also found that 
these understandings improve with learning opportunities. Thinking about 
models in science (as a learner or a scientist) demands a flexible commitment 
to the model that one holds. One needs to be able to view the model as a ten- 
tative explanation until a more fully explanatory one comes along. This can 
be hard for students, who often hold robust, alternative conceptions for 
many science concepts (e.g., Driver et al. 1985). The research on confirmation 
bias underscores the fact that people do not naturally consider rival models 
(Driver et al. 1996; Grosslight et al. 1991). Generating rival models from the 
outset is one way to encourage flexible commitment and deep consideration 
of the model in light of the available evidence (Grotzer 2002) and promises 
to lead to better evaluation of evidence. 
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Infusing a Focus on Causality in Science Learning 


When reasoning about complex phenomena, students rely on a series of 
default assumptions that often distort the nature of the causality involved. 
For instance, students often give linear or narrative explanations that are 
story-like: “first this happened, then it made that happen, and so on” Such 
explanations have a domino-like quality to them that actually is absent from 
many science concepts. Concepts such as symbiosis, pressure or density dif- 
ferentials, and electrical circuits are distinctly nonlinear in form. They involve 
mutual, relational, or cyclic patterns (Grotzer 2003). Concepts might appear 
straightforward but reveal complexity as soon as one dives below the surface. 
In addition to nonlinear patterns, they may include nonobvious causes; time 
delays and spatial gaps between causes and effects; distributed, nonintention- 
al agency; and probabilistic causation where the level of correspondence 
between causes and effects varies. Abrahamson and Wilensky (2005) found 
that many of the heuristics necessary for reasoning about complex systems 
run counter to those involved in reasoning about the linear systems with 
which students appear to be more familiar. 

Causal default assumptions impact what evidence people attend to and 
the salience that they attach to it. For example, when spatial gaps and time 
delays are present, people are less likely to notice relevant evidence. Instances 
of distributed causal agency, where many people acting on one level con- 
tribute to an emergent outcome on another level, are difficult to analyze. For 
example, in the case of global warming, recognizing our individual contribu- 
tions to the problem (and thus what we might do about them) is extremely 
difficult because evidence of the problem is available only at the level of the 
collective outcome. 

Increasingly, science education research is considering how to help stu- 
dents learn to think about causality in more complex ways. Students need 
explicit opportunities to reflect on their default patterns, learn the new causal 
patterns, and observe how the latter do a better job explaining the phenome- 
non at hand. Offering rich opportunities to learn complex causal concepts 
where students have the opportunity to discover different phenomena has 
shown some promise. Wilensky and Resnick (e.g., Resnick 1996; Wilensky 
1998; Wilensky and Resnick 1999) have used multiagent modeling in numer- 
ous studies on the concepts of emergence. They have demonstrated that con- 
structionist opportunities to work with dynamic, object-based models that 
reveal complex causal concepts result in new insights into the nature of com- 
plex phenomena such as gas laws and the behavior of slime molds. Chiu and 
colleagues (2002) found that having mentors unpack their thinking about 
complex problems that are just beyond the independent ability of the stu- 
dents made a significant difference in the students’ understanding of the con- 
cepts of simultaneity and randomness as these relate to chemical equilibrium 
as compared to students in a control group. Grotzer and colleagues (e.g., 
Grotzer and Basca 2003; Grotzer 2000; Perkins and Grotzer 2005) contrasted 
three pedagogical conditions across a number of science concepts (electricity, 
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ecosystems, air pressure, and density) that engaged students in thinking 
about simultaneity, multiple causes and effects, nonobvious causes, nonlin- 
earity, and outcomes due to balance or imbalance between multiple variables. 
They found that a combination of activities designed to reveal the underlying 
causal concepts and explicit discussion about the nature of causality (what is 
hard to grasp about it, how particular causal patterns differ from other causal 
patterns) led to deeper understanding. These students significantly outper- 
formed students who participated only in the causally focused activities or 
who did not participate in the causally focused activities or discussion but did 
have “best practices” science units that included extensive model building by 
students, evaluating evidence, Socratic discussion, dynamic computer mod- 
els, and attention to students’ evolving models. The differences were especial- 
ly dramatic for science concepts where the causality was least linear, sequen- 
tial, and direct. 


Beyond Teaching Empiricism 

Osborne (2002) has argued that we need to move beyond a focus on empiri- 
cism when thinking about how students analyze evidence. A majority of the 
public interacts with science text in popular accounts or journalistic versions. 
Korpan and colleagues (1997) have argued that media reports of scientific 
research are a pervasive and important source of new scientific knowledge 
and that the ability to evaluate conclusions found in those reports is an 
important form of scientific literacy. 

Korpan and colleagues (1997) found that students generated a range of 
requests for information and that they focused most often on ow the 
research was conducted and why the results might have occurred. They made 
fewer requests for other types of information, such as what was found, who 
conducted the research, where it was conducted, or whether related research 
had been conducted. They seemed most influenced by the plausibility of the 
conclusions, typicality of the phenomena, and their own personal familiarity 
with the phenomena. In a series of studies conducted with his graduate stu- 
dents, Chinn (Buckland and Chinn 2007; DiFranco and Chinn 2007; Hung 
and Chinn 2007; Pluta and Chinn 2007) has also explored how students 
make sense of evidence in published studies and how they coordinate that 
data across studies. Pluta and Chinn (2007) considered how seventh graders 
resolved conflicting pairs of news stories. The stories were similar to those 
published in newspaper accounts but had a slightly greater emphasis on 
methodology. The studies focused on scientific cases (deformed frogs, 
dinosaur metabolism, aspartame, etc.) that students were given information 
about. Students primarily reasoned at the level of explanations. However, 
students often had difficulty connecting the study details to the specific expla- 
nation. They ignored study details that might have helped them to integrate 
the evidence and their explanation, as well as details that did not fit with their 
initial ideas. Direct replication failure was genuinely baffling to students. 
While few students were able to use the details of the study to coordinate the 
results, they revealed a wide range of strategies across the groups of students. 


TEACHING EVIDENCE-BASED DECISION MAKING 


Ninety percent of students used more than one strategy (such as method- 
ological differences, bias, chance, etc.) to account for the conflicting scientific 
interpretations within a given problem and 70 percent of students used more 
than one strategy across different problem types. This is promising, particu- 
larly in classrooms where students engage in discussion about evidence. Pluta 
and Chinn (2007) found that certain forms of reasoning were less common, 
such as reasoning about bias or chance. These concepts warrant additional 
attention to help students understand and apply them. In general, these stud- 
ies stress the need for authentic practice in helping students learn how to 
apply strategies in particular instances. 


SUGGESTIONS FOR NEXT STEPS AND NEW DIRECTIONS 
TO IMPROVE SCIENTIFIC EVIDENTIAL REASONING 


Scientific evidential reasoning is a challenging yet important problem space. 
We have much to build on in taking next steps, and some clear needs are 
present that suggest directions for our collective efforts in improving the sci- 
entific literacy of the general population through education. 

A fairly extensive and informative research literature exists; it raises many 
puzzles but also offers solid information about approaches that are most 
promising. We have the most information about how students reason in the 
context of inquiry-based learning. The existing standards offer a strong basis 
from which to work. They draw solidly from the research base and make use- 
ful recommendations to classroom teachers. The focus, however, is largely on 
experimentation. Standards that include analyzing research evidence from 
published sources would promote greater scientific literacy even among those 
who do not plan to become scientists. 

Far less research has looked at how students of different ages reason about 
published scientific reports of real world studies (Chinn and Malhotra 2002). 
Chinn and colleagues (e.g., Hung and Chinn, 2007; Pluta and Chinn 2007) 
have taken some important first steps here. More research focused on inter- 
preting findings is needed. Continued exploration of the barriers to under- 
standing is also needed so that we can find approaches that offer the most 
leverage in solving the problem. For instance, work focused on helping stu- 
dents examine their causal default assumptions may systematically impact 
how they generate explanations. Similarly, helping students understand the 
epistemology of science helps them to view science as an endeavor differently 
and to realize that it is more than just learning facts. 

A better bridge between research and practice is needed. While a fair 
amount is known about evidence evaluation skills and the epistemological 
knowledge that supports them, these findings have been slow to make their 
way into mainstream practice. This lack of an effective bridge is a perennial 
problem in applying research findings. If future investigations were more 
closely situated at the intersection of practice and basic research on learning— 
in “Pasteur’s Quadrant” (Stokes 1997) —a bridge might no longer be neces- 
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sary. We need accessible pedagogies that build upon the research results. The 
kind of work that Sandoval and colleagues (e.g., Sandoval and Reiser 2004) 
are engaged in—designing computer programs that teach the epistemology 
of science—holds promise here. 

Probably the biggest hurdle, however, has to do with supporting teach- 
ers. To assume that teachers will be able to teach the epistemology of science 
in all its nuances without a considerable investment in professional develop- 
ment and instructional materials is irrationally optimistic. The type of 
nuanced understanding called for in the science education research comes 
from a deep understanding of scientific inquiry and the expertise associated 
with having opportunities to engage in science. This stands in sharp contrast 
to the comfort level that many teachers, especially at the elementary level, 
have with science and the high rate of teacher turnover, particularly in urban 
schools. 

What is obvious, even from this brief overview, is how challenging the 
problem space of scientific reasoning and education is. However, the pay-off 
from engaging with this problem space is a scientifically literate population: 
critical consumers of the abundance of available scientific information who 
can reason effectively about evidence and a generation of scientists capable of 
understanding a complex, dynamic world. Substantial resources already exist 
to support these efforts. The imperative is clear. 
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